[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:12 dist-test-slave-tzgx krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:12 dist-test-slave-tzgx krb5kdc[399](info): set up 2 sockets
Sep 27 12:35:12 dist-test-slave-tzgx krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 12:35:13.752849 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:13 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:13.876058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:13.993510 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:13.993795 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:13.993837 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:13.997972 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:13.998039 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:13.998056 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:13.998072 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:13.998092 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:14.003587 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:14.004709 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:14.005748 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:14.015445 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.015604 412 server_base.cc:1034] running on GCE node
W20260927 12:35:14.015427 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:14.015489 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.016323 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:14.017486 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:14.018680 412 hybrid_clock.cc:648] HybridClock initialized: now 1790512514018624 us; error 72 us; skew 500 ppm
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:14.021903 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:14.022945 412 webserver.cc:460] Webserver started at http://127.0.98.62:42061/ using document root <none> and password file <none>
I20260927 12:35:14.023581 412 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:14.023646 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:14.023854 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:14.026038 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "06d9caa935eab51c9db474857cd4a6ec"
server_key_iv: "295206889634ee9186c535704c7a8380"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.026615 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "06d9caa935eab51c9db474857cd4a6ec"
server_key_iv: "295206889634ee9186c535704c7a8380"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.030607 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 12:35:14.033320 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.034626 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:14.034754 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "06d9caa935eab51c9db474857cd4a6ec"
server_key_iv: "295206889634ee9186c535704c7a8380"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.034866 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:14.047633 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:14.050767 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:14.050972 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:14.058198 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33835
I20260927 12:35:14.058236 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33835 every 8 connection(s)
I20260927 12:35:14.059139 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:14.061997 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:14.062772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 412
I20260927 12:35:14.062932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:14.064692 392 external_mini_cluster.cc:1428] Setting key 2cf3e0831fc09f36b79e5eaf56fe8cc6
I20260927 12:35:14.067890 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92: Bootstrap starting.
I20260927 12:35:14.070401 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:14.071146 480 log.cc:824] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:14.073410 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92: No bootstrap required, opened a new log
I20260927 12:35:14.077817 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 12:35:14.078132 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:14.078195 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c1f78bd32a4b6b87a2a6aed52a4d92, State: Initialized, Role: FOLLOWER
I20260927 12:35:14.078727 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 12:35:14.078895 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:14.078970 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:14.079072 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:14.080349 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 12:35:14.080848 480 leader_election.cc:302] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00c1f78bd32a4b6b87a2a6aed52a4d92; no voters:
I20260927 12:35:14.081195 480 leader_election.cc:288] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:14.081436 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:14.081736 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [term 1 LEADER]: Becoming Leader. State: Replica: 00c1f78bd32a4b6b87a2a6aed52a4d92, State: Running, Role: LEADER
I20260927 12:35:14.082053 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } }
I20260927 12:35:14.082612 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:14.083618 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } } }
I20260927 12:35:14.083729 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:14.084007 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c1f78bd32a4b6b87a2a6aed52a4d92. Latest consensus state: current_term: 1 leader_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c1f78bd32a4b6b87a2a6aed52a4d92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33835 } } }
I20260927 12:35:14.084080 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:14.084350 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:14.088146 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:14.090571 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.067813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.98.62:33835)
0927 12:35:14.068261 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:14.068276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:14.069357 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 12:35:14.072593 (+ 3236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:14.072609 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:14.072933 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:14.073217 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:14.076241 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.077211 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.078961 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.079520 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.079652 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.084643 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:14.084650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:14.084668 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:14.084705 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:14.086673 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.087215 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.087222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.087228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.087373 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.087735 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.087739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.087741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.088303 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:14.088509 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:14.089334 (+ 825us) server_negotiation.cc:299] Negotiation successful
0927 12:35:14.089625 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":111,"threads_started":1}
I20260927 12:35:14.094487 494 catalog_manager.cc:1347] Generated new cluster ID: 2dc7f3d847e244298ba972c76ff889f1
I20260927 12:35:14.094551 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:14.104388 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:14.105291 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:14.111047 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00c1f78bd32a4b6b87a2a6aed52a4d92: Generated new TSK 0
I20260927 12:35:14.111761 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:14.142869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:14.282054 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:14.282342 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:14.282395 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:14.286705 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:14.286784 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:14.286849 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:14.292397 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:14.293471 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:14.294477 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:14.304489 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:14.304489 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:14.304517 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.304715 506 server_base.cc:1034] running on GCE node
I20260927 12:35:14.305158 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:14.305858 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:14.307027 506 hybrid_clock.cc:648] HybridClock initialized: now 1790512514307007 us; error 39 us; skew 500 ppm
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:14.309937 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:14.310904 506 webserver.cc:460] Webserver started at http://127.0.98.1:44453/ using document root <none> and password file <none>
I20260927 12:35:14.311494 506 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:14.311563 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:14.311765 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:14.314007 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0d850dc35a7474dbda31beae96059b9"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "701aff4d8385fd105465327656facbd3"
server_key_iv: "bf7c5d9ad8b4ab179f7083d5f512cba6"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.314611 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d850dc35a7474dbda31beae96059b9"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "701aff4d8385fd105465327656facbd3"
server_key_iv: "bf7c5d9ad8b4ab179f7083d5f512cba6"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.318706 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:14.321743 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.323187 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:14.323335 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "e0d850dc35a7474dbda31beae96059b9"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "701aff4d8385fd105465327656facbd3"
server_key_iv: "bf7c5d9ad8b4ab179f7083d5f512cba6"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.323477 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:14.346222 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:14.349956 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:14.350174 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:14.350900 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:14.351840 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:14.351898 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.351958 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:14.351991 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.361384 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42841
I20260927 12:35:14.361406 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42841 every 8 connection(s)
I20260927 12:35:14.362334 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:14.370983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 506
I20260927 12:35:14.371084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:14.371323 392 external_mini_cluster.cc:1428] Setting key 5a30d567a9afd73a7e4f185c7cd0e1f9
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:14.378283 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.364284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44959 (local address 127.0.98.62:33835)
0927 12:35:14.364441 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:14.364446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:14.365407 (+ 961us) server_negotiation.cc:407] Connection header received
0927 12:35:14.366781 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:14.366784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:14.366847 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:14.366942 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:14.368500 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.369094 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.370369 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.370621 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.370736 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.373612 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:14.373616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:14.373619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:14.373651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:14.375449 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.376381 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.376385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.376387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.376493 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.376916 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.376919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.376922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.377199 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:14.377305 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:14.377931 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 12:35:14.378099 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:35:14.379194 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.364719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33835 (local address 127.0.98.1:44959)
0927 12:35:14.365231 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:14.365280 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:14.366532 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:14.367176 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:14.367185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:14.367585 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:14.368308 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:14.368329 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.369247 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.369250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:14.370225 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:14.370234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.370789 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.370792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:14.370959 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.371644 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:14.371663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:14.373421 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:14.375599 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:14.375605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:14.375620 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:14.376244 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:14.376622 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:14.376625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:14.376626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:14.376809 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:14.377326 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:14.377333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:14.377646 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 12:35:14.377824 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 12:35:14.378157 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:35:14.381035 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33835
I20260927 12:35:14.381285 635 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:14.381841 635 heartbeater.cc:511] Master 127.0.98.62:33835 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:35:14.383558 444 ts_manager.cc:194] Registered new tserver with Master: e0d850dc35a7474dbda31beae96059b9 (127.0.98.1:42841)
I20260927 12:35:14.385084 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:14.399360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:14.526316 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:14.526607 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:14.526661 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:14.530819 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:14.530893 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:14.530962 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:14.536805 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:14.537914 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:14.538993 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:14.548614 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.548734 640 server_base.cc:1034] running on GCE node
W20260927 12:35:14.548676 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:14.548676 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.549271 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:14.549922 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:14.551082 640 hybrid_clock.cc:648] HybridClock initialized: now 1790512514551064 us; error 40 us; skew 500 ppm
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:14.553942 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:14.554924 640 webserver.cc:460] Webserver started at http://127.0.98.2:34447/ using document root <none> and password file <none>
I20260927 12:35:14.555480 640 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:14.555549 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:14.555773 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:14.557899 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "74252b4428324deeb4c5289985895cb9"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "16854a2ec0330039f4aeaade3ce674dc"
server_key_iv: "a1358bb70b95774cec38a682a210ac1f"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.558487 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74252b4428324deeb4c5289985895cb9"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "16854a2ec0330039f4aeaade3ce674dc"
server_key_iv: "a1358bb70b95774cec38a682a210ac1f"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.562439 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:14.565107 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.566468 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:14.566589 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "74252b4428324deeb4c5289985895cb9"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "16854a2ec0330039f4aeaade3ce674dc"
server_key_iv: "a1358bb70b95774cec38a682a210ac1f"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.566704 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:14.576687 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:14.579916 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:14.580155 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:14.580775 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:14.581632 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:14.581682 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.581735 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:14.581763 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.591156 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46703
I20260927 12:35:14.591204 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46703 every 8 connection(s)
I20260927 12:35:14.592121 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:14.595683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 640
I20260927 12:35:14.595798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:14.596033 392 external_mini_cluster.cc:1428] Setting key 3caf6004ea192a13de8480f416cc5ef6
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:14.606318 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.594025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53875 (local address 127.0.98.62:33835)
0927 12:35:14.594191 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:14.594195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:14.595115 (+ 920us) server_negotiation.cc:407] Connection header received
0927 12:35:14.596541 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:14.596544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:14.596603 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:14.596680 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:14.597938 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.598459 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.599612 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.599885 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.599980 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.602847 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:14.602851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:14.602853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:14.602879 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:14.604257 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.604837 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.604840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.604842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.604922 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.605252 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.605255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.605257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.605471 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:14.605554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:14.606083 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 12:35:14.606196 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:35:14.607312 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.594431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33835 (local address 127.0.98.2:53875)
0927 12:35:14.594932 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:14.594976 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:14.596304 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:14.596872 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:14.596880 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:14.597264 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:14.597793 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:14.597809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.598587 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.598590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:14.599496 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:14.599505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.600051 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.600062 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:14.600283 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.600845 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:14.600863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:14.602643 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:14.604365 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:14.604371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:14.604390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:14.604745 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:14.605001 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:14.605003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:14.605004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:14.605145 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:14.605564 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:14.605570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:14.605835 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 12:35:14.606000 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 12:35:14.606271 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260927 12:35:14.609158 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33835
I20260927 12:35:14.609498 769 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:14.610117 769 heartbeater.cc:511] Master 127.0.98.62:33835 requested a full tablet report, sending...
I20260927 12:35:14.611339 444 ts_manager.cc:194] Registered new tserver with Master: 74252b4428324deeb4c5289985895cb9 (127.0.98.2:46703)
I20260927 12:35:14.612157 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:14.622902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:14.735783 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:14.736073 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:14.736146 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:14.740346 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:14.740413 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:14.740477 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:14.745820 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:14.746840 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:14.747852 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:14.757473 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.757704 774 server_base.cc:1034] running on GCE node
W20260927 12:35:14.757479 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:14.757592 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:14.758217 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:14.758872 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:14.760018 774 hybrid_clock.cc:648] HybridClock initialized: now 1790512514759992 us; error 43 us; skew 500 ppm
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:14.762785 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:14.763728 774 webserver.cc:460] Webserver started at http://127.0.98.3:45883/ using document root <none> and password file <none>
I20260927 12:35:14.764333 774 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:14.764398 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:14.764623 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:14.766763 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "e747a57bda404d6faa2653c86262ff01"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "bf250d542489f5e1da21460fcda07921"
server_key_iv: "ed6efa7bc65ec41f6a01f6237188e680"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.767344 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e747a57bda404d6faa2653c86262ff01"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "bf250d542489f5e1da21460fcda07921"
server_key_iv: "ed6efa7bc65ec41f6a01f6237188e680"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.771487 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:14.774207 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.775599 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:14.775707 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "e747a57bda404d6faa2653c86262ff01"
format_stamp: "Formatted at 2026-09-27 12:35:14 on dist-test-slave-tzgx"
server_key: "bf250d542489f5e1da21460fcda07921"
server_key_iv: "ed6efa7bc65ec41f6a01f6237188e680"
server_key_version: "encryptionkey@0"
I20260927 12:35:14.775833 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:14.793504 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:14.796802 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:14.796976 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:14.797608 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:14.798585 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:14.798636 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.798681 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:14.798710 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:14.808766 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34333
I20260927 12:35:14.808849 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34333 every 8 connection(s)
I20260927 12:35:14.809727 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:14.819425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 774
I20260927 12:35:14.819517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:14.819761 392 external_mini_cluster.cc:1428] Setting key 950f277e0ea3dfcbf00b6c25e78a530b
I20260927 12:35:14.823201 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.811582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49949 (local address 127.0.98.62:33835)
0927 12:35:14.811719 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:14.811723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:14.812629 (+ 906us) server_negotiation.cc:407] Connection header received
0927 12:35:14.813939 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:14.813943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:14.814008 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:14.814083 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:14.815288 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.815816 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.817005 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.817220 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.817309 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.819629 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:14.819632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:14.819634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:14.819659 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:14.821310 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.821771 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.821774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.821776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.821865 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.822163 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.822167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.822169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.822350 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:14.822433 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:14.822915 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 12:35:14.823049 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:35:14.824038 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.811987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33835 (local address 127.0.98.3:49949)
0927 12:35:14.812477 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:14.812521 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:14.813728 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:14.814273 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:14.814280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:14.814646 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:14.815140 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:14.815156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.815935 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.815938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:14.816881 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:14.816890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.817357 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.817360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:14.817528 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.818070 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:14.818085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:14.819426 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:14.821405 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:14.821412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:14.821427 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:14.821679 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:14.821943 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:14.821946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:14.821948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:14.822080 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:14.822440 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:14.822447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:14.822692 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 12:35:14.822861 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 12:35:14.823236 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":112,"threads_started":1}
I20260927 12:35:14.825156 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33835
I20260927 12:35:14.825361 903 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:14.825798 903 heartbeater.cc:511] Master 127.0.98.62:33835 requested a full tablet report, sending...
I20260927 12:35:14.826738 444 ts_manager.cc:194] Registered new tserver with Master: e747a57bda404d6faa2653c86262ff01 (127.0.98.3:34333)
I20260927 12:35:14.827308 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49949
I20260927 12:35:14.834108 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:14.859334 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.850295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.0.98.1:42841)
0927 12:35:14.850583 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:14.850596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:14.850620 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:35:14.850712 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:14.850717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:14.850908 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:14.851032 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:14.851603 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.852403 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.853217 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.853612 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.853696 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.855979 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:14.855988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:14.856002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:14.856025 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:14.857418 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.857783 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.857788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.857789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.857860 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.858043 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.858045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.858046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.858268 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:14.858421 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:14.858564 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 12:35:14.858688 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
W20260927 12:35:14.860342 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:14.884859 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:14.876670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.0.98.1:42841)
0927 12:35:14.876844 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:14.876848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:14.876896 (+ 48us) server_negotiation.cc:407] Connection header received
0927 12:35:14.877026 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:14.877029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:14.877073 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:14.877135 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:14.877791 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.878320 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.879222 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:14.879449 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:14.879528 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:14.882024 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:14.882027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:14.882030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:14.882056 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:14.883288 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.883658 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.883661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.883662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.883729 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:14.884081 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:14.884085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:14.884124 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:14.884294 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:14.884394 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:14.884597 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 12:35:14.884699 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 12:35:14.885842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 506
I20260927 12:35:14.891656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 640
I20260927 12:35:14.898108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 774
I20260927 12:35:14.902907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 412
2026-09-27T12:35:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2128 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[917](info): set up 2 sockets
Sep 27 12:35:14 dist-test-slave-tzgx krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 12:35:15.859645 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:15 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:15.979924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:16.089134 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:16.089442 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:16.089484 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:16.093636 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:16.093701 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:16.093720 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:16.093735 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:16.093750 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:16.099262 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:16.100387 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:16.101406 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:16.109997 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:16.110054 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:16.110174 930 server_base.cc:1034] running on GCE node
W20260927 12:35:16.110040 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:16.110842 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:16.111958 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:16.113152 930 hybrid_clock.cc:648] HybridClock initialized: now 1790512516113120 us; error 44 us; skew 500 ppm
Sep 27 12:35:16 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:16.115970 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:16.116909 930 webserver.cc:460] Webserver started at http://127.0.98.62:33045/ using document root <none> and password file <none>
I20260927 12:35:16.117456 930 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:16.117530 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:16.117726 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:16.119872 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "eb43fe3e5f5244c09923297e52dfed26"
format_stamp: "Formatted at 2026-09-27 12:35:16 on dist-test-slave-tzgx"
server_key: "2b06f825001104981a87bc888acf0a2b"
server_key_iv: "9a516ccf4559650b81af629b780a8333"
server_key_version: "encryptionkey@0"
I20260927 12:35:16.120497 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb43fe3e5f5244c09923297e52dfed26"
format_stamp: "Formatted at 2026-09-27 12:35:16 on dist-test-slave-tzgx"
server_key: "2b06f825001104981a87bc888acf0a2b"
server_key_iv: "9a516ccf4559650b81af629b780a8333"
server_key_version: "encryptionkey@0"
I20260927 12:35:16.124396 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:16.127018 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:16.128381 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:16.128509 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "eb43fe3e5f5244c09923297e52dfed26"
format_stamp: "Formatted at 2026-09-27 12:35:16 on dist-test-slave-tzgx"
server_key: "2b06f825001104981a87bc888acf0a2b"
server_key_iv: "9a516ccf4559650b81af629b780a8333"
server_key_version: "encryptionkey@0"
I20260927 12:35:16.128621 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:16.143186 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:16.146944 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:16.147149 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:16.154026 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45789
I20260927 12:35:16.154076 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45789 every 8 connection(s)
I20260927 12:35:16.155046 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:16.155952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 930
I20260927 12:35:16.156082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:16.156345 392 external_mini_cluster.cc:1428] Setting key 012cd20f2a3b2eb230ad96a2a0e52001
I20260927 12:35:16.158303 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:16.164750 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26: Bootstrap starting.
Sep 27 12:35:16 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:16.167171 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:16.167891 998 log.cc:824] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:16.170213 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26: No bootstrap required, opened a new log
I20260927 12:35:16.170579 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:16.157639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.0.98.62:45789)
0927 12:35:16.158033 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:16.158049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:16.158086 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:35:16.158769 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:16.158787 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:16.159129 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:16.159409 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:16.160257 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:16.161264 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:16.162379 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:16.162874 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:16.163002 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:16.165808 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:16.165814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:16.165827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:16.165859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:16.167437 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:16.167882 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:16.167887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:16.167892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:16.168011 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:16.168348 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:16.168351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:16.168352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:16.168884 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:16.169035 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:16.169430 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 12:35:16.169753 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260927 12:35:16.174064 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb43fe3e5f5244c09923297e52dfed26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260927 12:35:16.174310 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:16.174374 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb43fe3e5f5244c09923297e52dfed26, State: Initialized, Role: FOLLOWER
I20260927 12:35:16.174786 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb43fe3e5f5244c09923297e52dfed26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260927 12:35:16.174916 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:16.174980 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:16.175086 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:16.176055 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb43fe3e5f5244c09923297e52dfed26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260927 12:35:16.176446 998 leader_election.cc:302] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb43fe3e5f5244c09923297e52dfed26; no voters:
I20260927 12:35:16.176766 998 leader_election.cc:288] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:16.176909 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:16.177143 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [term 1 LEADER]: Becoming Leader. State: Replica: eb43fe3e5f5244c09923297e52dfed26, State: Running, Role: LEADER
I20260927 12:35:16.177522 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb43fe3e5f5244c09923297e52dfed26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } }
I20260927 12:35:16.178153 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:16.179315 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb43fe3e5f5244c09923297e52dfed26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb43fe3e5f5244c09923297e52dfed26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } } }
I20260927 12:35:16.179320 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb43fe3e5f5244c09923297e52dfed26. Latest consensus state: current_term: 1 leader_uuid: "eb43fe3e5f5244c09923297e52dfed26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb43fe3e5f5244c09923297e52dfed26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45789 } } }
I20260927 12:35:16.179468 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:16.179456 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:16.179874 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:16.183341 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:16.189494 1012 catalog_manager.cc:1347] Generated new cluster ID: b7e9c6c0b66d4492a53bed7063395e79
I20260927 12:35:16.189579 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:16.217825 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:16.219163 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:16.228941 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb43fe3e5f5244c09923297e52dfed26: Generated new TSK 0
I20260927 12:35:16.229902 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:16.261320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:16.373425 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:16.373733 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:16.373797 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:16.378139 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:16.378213 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:16.378294 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:16.384060 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:16.385286 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:16.386447 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:16.396687 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:16.397053 1024 server_base.cc:1034] running on GCE node
W20260927 12:35:16.396694 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:16.396740 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:16.398110 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:16.398792 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:16.399991 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790512516399962 us; error 40 us; skew 500 ppm
Sep 27 12:35:16 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:16.402837 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:16.403782 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43353/ using document root <none> and password file <none>
I20260927 12:35:16.404399 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:16.404469 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:16.404680 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:16.406800 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "c85c3ddd9cbf415f9d2012ac0af47653"
format_stamp: "Formatted at 2026-09-27 12:35:16 on dist-test-slave-tzgx"
server_key: "b985f73e9656c99a571d9f7e0f2b7ccf"
server_key_iv: "178093440bcb0e1070ab24dde878614f"
server_key_version: "encryptionkey@0"
I20260927 12:35:16.407378 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c85c3ddd9cbf415f9d2012ac0af47653"
format_stamp: "Formatted at 2026-09-27 12:35:16 on dist-test-slave-tzgx"
server_key: "b985f73e9656c99a571d9f7e0f2b7ccf"
server_key_iv: "178093440bcb0e1070ab24dde878614f"
server_key_version: "encryptionkey@0"
I20260927 12:35:16.411454 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:16.414067 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:16.415323 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:35:16.415446 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "c85c3ddd9cbf415f9d2012ac0af47653"
format_stamp: "Formatted at 2026-09-27 12:35:16 on dist-test-slave-tzgx"
server_key: "b985f73e9656c99a571d9f7e0f2b7ccf"
server_key_iv: "178093440bcb0e1070ab24dde878614f"
server_key_version: "encryptionkey@0"
I20260927 12:35:16.415556 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:16.448391 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:16.451253 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:16.451426 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:16.452030 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:16.452863 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:16.452917 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:16.452975 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:16.453003 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:16.461339 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41517
I20260927 12:35:16.461370 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41517 every 8 connection(s)
I20260927 12:35:16.462211 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:16.468477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1024
I20260927 12:35:16.468585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:16.468851 392 external_mini_cluster.cc:1428] Setting key 93afdd14bc7ce3b07d37b554250156e5
Sep 27 12:35:16 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:16.476784 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:16.463950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36971 (local address 127.0.98.62:45789)
0927 12:35:16.464118 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:16.464123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:16.464996 (+ 873us) server_negotiation.cc:407] Connection header received
0927 12:35:16.466276 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:16.466279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:16.466324 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:16.466395 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:16.467535 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:16.468076 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:16.469517 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:16.469808 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:16.469921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:16.472820 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:16.472823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:16.472825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:16.472852 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:16.474517 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:16.475098 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:16.475102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:16.475104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:16.475193 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:16.475544 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:16.475547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:16.475549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:16.475777 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:16.475868 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:16.476469 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 12:35:16.476612 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:35:16.477727 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:16.464370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45789 (local address 127.0.98.1:36971)
0927 12:35:16.464825 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:16.464868 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:16.466087 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:16.466550 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:16.466556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:16.466931 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:16.467405 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:16.467419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:16.468180 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:16.468182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:16.469405 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:16.469414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:16.469985 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:16.469989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:16.470177 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:16.470735 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:16.470752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:16.472601 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:16.474679 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:16.474685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:16.474702 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:16.474993 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:16.475281 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:16.475284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:16.475286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:16.475443 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:16.475884 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:16.475891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:16.476219 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 12:35:16.476411 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 12:35:16.476801 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:35:16.479045 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45789
I20260927 12:35:16.479296 1153 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:16.479851 1153 heartbeater.cc:511] Master 127.0.98.62:45789 requested a full tablet report, sending...
I20260927 12:35:16.481565 962 ts_manager.cc:194] Registered new tserver with Master: c85c3ddd9cbf415f9d2012ac0af47653 (127.0.98.1:41517)
I20260927 12:35:16.482827 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:35:17.485487 1153 heartbeater.cc:503] Master 127.0.98.62:45789 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 12:35:17.571933 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.101s user 0.002s sys 0.001s
I20260927 12:35:17.572501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:17.689034 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:17.689330 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:17.689394 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:17.693601 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:17.693673 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:17.693758 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:17.699632 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:17.700795 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:17.701915 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:17.711820 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:17.711843 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:17.712111 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:17.712445 1158 server_base.cc:1034] running on GCE node
I20260927 12:35:17.712944 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:17.713717 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:17.714888 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790512517714852 us; error 63 us; skew 500 ppm
Sep 27 12:35:17 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:17.717777 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:17.718729 1158 webserver.cc:460] Webserver started at http://127.0.98.2:44183/ using document root <none> and password file <none>
I20260927 12:35:17.719317 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:17.719390 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:17.719583 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:17.721732 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "dcdf2e0508334841bc9aefedd1dc49ea"
format_stamp: "Formatted at 2026-09-27 12:35:17 on dist-test-slave-tzgx"
server_key: "9a160c6718ec8ce3d00259bf7f75fe47"
server_key_iv: "0ec75730e578b06f65b4f438e82a2bc0"
server_key_version: "encryptionkey@0"
I20260927 12:35:17.722291 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dcdf2e0508334841bc9aefedd1dc49ea"
format_stamp: "Formatted at 2026-09-27 12:35:17 on dist-test-slave-tzgx"
server_key: "9a160c6718ec8ce3d00259bf7f75fe47"
server_key_iv: "0ec75730e578b06f65b4f438e82a2bc0"
server_key_version: "encryptionkey@0"
I20260927 12:35:17.726244 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:17.728925 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:17.730341 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:17.730463 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "dcdf2e0508334841bc9aefedd1dc49ea"
format_stamp: "Formatted at 2026-09-27 12:35:17 on dist-test-slave-tzgx"
server_key: "9a160c6718ec8ce3d00259bf7f75fe47"
server_key_iv: "0ec75730e578b06f65b4f438e82a2bc0"
server_key_version: "encryptionkey@0"
I20260927 12:35:17.730577 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:17.750258 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:17.753974 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:17.754175 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:17.754871 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:17.755913 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:17.755982 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:17.756034 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:17.756062 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:17.764639 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45415
I20260927 12:35:17.764695 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45415 every 8 connection(s)
I20260927 12:35:17.765565 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:17.769008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1158
I20260927 12:35:17.769107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:17.769336 392 external_mini_cluster.cc:1428] Setting key b03c264d32c6a6c9fa287395555fd46d
Sep 27 12:35:17 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:17.781746 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:17.767388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40277 (local address 127.0.98.62:45789)
0927 12:35:17.767657 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:17.767662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:17.768566 (+ 904us) server_negotiation.cc:407] Connection header received
0927 12:35:17.770257 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:17.770263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:17.770343 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:17.770444 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:17.773392 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:17.773954 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:17.775215 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:17.775447 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:17.775537 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:17.778283 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:17.778287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:17.778290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:17.778338 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:17.779688 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:17.780203 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:17.780207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:17.780209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:17.780299 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:17.780637 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:17.780640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:17.780641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:17.780823 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:17.780906 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:17.781423 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 12:35:17.781553 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":58,"threads_started":1}
I20260927 12:35:17.782572 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:17.767885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45789 (local address 127.0.98.2:40277)
0927 12:35:17.768405 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:17.768454 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:17.769932 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:17.770818 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:17.770829 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:17.772657 (+ 1828us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:17.773219 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:17.773235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:17.774089 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:17.774091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:17.775086 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:17.775097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:17.775609 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:17.775613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:17.775787 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:17.776473 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:17.776490 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:17.778106 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:17.779786 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:17.779792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:17.779811 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:17.780074 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:17.780384 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:17.780387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:17.780390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:17.780541 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:17.780918 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:17.780926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:17.781183 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 12:35:17.781351 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 12:35:17.781705 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":154,"threads_started":1}
I20260927 12:35:17.783733 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45789
I20260927 12:35:17.784039 1287 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:17.784588 1287 heartbeater.cc:511] Master 127.0.98.62:45789 requested a full tablet report, sending...
I20260927 12:35:17.785718 962 ts_manager.cc:194] Registered new tserver with Master: dcdf2e0508334841bc9aefedd1dc49ea (127.0.98.2:45415)
I20260927 12:35:17.786329 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:35:18.788473 1287 heartbeater.cc:503] Master 127.0.98.62:45789 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 12:35:18.985805 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.215s user 0.001s sys 0.002s
I20260927 12:35:18.986303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:19.100164 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:19.100442 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:19.100494 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:19.104650 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:19.104720 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:19.104784 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:19.110198 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:19.111310 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:19.112394 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:19.122605 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:19.122556 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:19.122556 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:19.122931 1293 server_base.cc:1034] running on GCE node
I20260927 12:35:19.123386 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:19.124151 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:19.125339 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790512519125323 us; error 36 us; skew 500 ppm
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:19.132547 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:19.133667 1293 webserver.cc:460] Webserver started at http://127.0.98.3:46573/ using document root <none> and password file <none>
I20260927 12:35:19.134259 1293 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:19.134337 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:19.134538 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:19.136894 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "7de9b4c531e148c3b2b68f576c18092d"
format_stamp: "Formatted at 2026-09-27 12:35:19 on dist-test-slave-tzgx"
server_key: "efea09324fc828a187a177c7f981e933"
server_key_iv: "32a3fae52bc0e96a9b2a2def04f3c68e"
server_key_version: "encryptionkey@0"
I20260927 12:35:19.137457 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7de9b4c531e148c3b2b68f576c18092d"
format_stamp: "Formatted at 2026-09-27 12:35:19 on dist-test-slave-tzgx"
server_key: "efea09324fc828a187a177c7f981e933"
server_key_iv: "32a3fae52bc0e96a9b2a2def04f3c68e"
server_key_version: "encryptionkey@0"
I20260927 12:35:19.141455 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 12:35:19.144179 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:19.145527 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:19.145653 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "7de9b4c531e148c3b2b68f576c18092d"
format_stamp: "Formatted at 2026-09-27 12:35:19 on dist-test-slave-tzgx"
server_key: "efea09324fc828a187a177c7f981e933"
server_key_iv: "32a3fae52bc0e96a9b2a2def04f3c68e"
server_key_version: "encryptionkey@0"
I20260927 12:35:19.145780 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:19.172340 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:19.175262 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:19.175426 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:19.176019 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:19.176834 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:19.176880 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:19.176920 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:19.176957 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:19.186213 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44103
I20260927 12:35:19.186260 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44103 every 8 connection(s)
I20260927 12:35:19.187201 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:19.192940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1293
I20260927 12:35:19.193070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:19.193346 392 external_mini_cluster.cc:1428] Setting key c5c0231865e2028bad8b5dedd3abc319
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:19.202020 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.189076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39011 (local address 127.0.98.62:45789)
0927 12:35:19.189336 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.189340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.190140 (+ 800us) server_negotiation.cc:407] Connection header received
0927 12:35:19.191557 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.191560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.191610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.191709 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:19.193007 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.193750 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.195186 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.195462 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.195553 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.198126 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:19.198129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:19.198132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:19.198158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:19.199643 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.200414 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.200418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.200421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.200506 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.200860 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.200862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.200864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.201056 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:19.201146 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.201702 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.201836 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260927 12:35:19.202925 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.189536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45789 (local address 127.0.98.3:39011)
0927 12:35:19.189984 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:19.190027 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:19.191324 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:19.191879 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:19.191886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:19.192320 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:19.192842 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.192856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.193882 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.193885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.195074 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.195085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.195614 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.195617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.195778 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.196392 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:19.196410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:19.197936 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:19.199777 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:19.199784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:19.199806 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:19.200157 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:19.200594 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:19.200597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:19.200599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:19.200761 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:19.201175 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:19.201182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:19.201456 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 12:35:19.201632 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 12:35:19.201990 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260927 12:35:19.204298 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45789
I20260927 12:35:19.204567 1422 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:19.205092 1422 heartbeater.cc:511] Master 127.0.98.62:45789 requested a full tablet report, sending...
I20260927 12:35:19.206142 962 ts_manager.cc:194] Registered new tserver with Master: 7de9b4c531e148c3b2b68f576c18092d (127.0.98.3:44103)
I20260927 12:35:19.206713 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39011
I20260927 12:35:19.207319 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:35:19.219827 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.210941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.0.98.62:45789)
0927 12:35:19.211116 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.211121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.211286 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:35:19.211379 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.211381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.211425 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.211501 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:19.212482 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.213052 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.213892 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.214287 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.214404 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.216220 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:19.216224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:19.216228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:19.216266 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:19.218162 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.218637 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.218640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.218641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.218726 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.218960 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.218962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.218963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.219192 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:19.219326 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.219539 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.219663 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:35:19.225610 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:19.228974 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:19.244339 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.238622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45415 (local address 127.0.0.1:48096)
0927 12:35:19.239072 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:19.239124 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:19.239260 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:19.239844 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:19.239851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:19.239877 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:19.240016 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.240023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.241221 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.241224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.242586 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.242595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.243707 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.243710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.243828 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.243872 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 12:35:19.243990 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 12:35:19.244212 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260927 12:35:19.244773 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.238828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.0.98.2:45415)
0927 12:35:19.239138 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.239145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.239199 (+ 54us) server_negotiation.cc:407] Connection header received
0927 12:35:19.239431 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.239437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.239679 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.239846 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:19.240155 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.241105 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.242703 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.243388 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.243484 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.243650 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.244538 (+ 888us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.244655 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260927 12:35:19.245213 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.239591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41517 (local address 127.0.0.1:44524)
0927 12:35:19.240130 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:19.240143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:19.240223 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:19.240878 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:19.240881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:19.240901 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:19.241001 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.241007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.242472 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.242476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.243692 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.243702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.244869 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.244873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.244967 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.244992 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:35:19.245046 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 12:35:19.245115 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":87,"threads_started":1}
I20260927 12:35:19.245929 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.239280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44103 (local address 127.0.0.1:49472)
0927 12:35:19.239487 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:19.239496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:19.239561 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:19.241532 (+ 1971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:19.241534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:19.241550 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:19.241746 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.241754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.242950 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.242953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.244061 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.244070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.245566 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.245570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.245655 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.245679 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:35:19.245739 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:35:19.245808 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260927 12:35:19.246021 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.239989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.98.3:44103)
0927 12:35:19.241188 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.241197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.241214 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:35:19.241272 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.241276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.241409 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.241536 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:19.241852 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.242793 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.244228 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.245465 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.245571 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.245711 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.245766 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.245862 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":123,"threads_started":1}
I20260927 12:35:19.246064 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.239742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.0.98.1:41517)
0927 12:35:19.240445 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.240451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.240472 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:35:19.240534 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.240538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.240743 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.240893 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:19.241142 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.242348 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.243836 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.244740 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.245669 (+ 929us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.245804 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.245847 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.245940 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":143,"threads_started":1}
I20260927 12:35:19.248003 1221 tablet_service.cc:1467] Processing CreateTablet for tablet aaf1e6822ff14407b9f6270ef1515b29 (DEFAULT_TABLE table=test-table [id=6370281f312b414b9c069214f8e4eb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:19.249218 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf1e6822ff14407b9f6270ef1515b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:19.251169 1087 tablet_service.cc:1467] Processing CreateTablet for tablet aaf1e6822ff14407b9f6270ef1515b29 (DEFAULT_TABLE table=test-table [id=6370281f312b414b9c069214f8e4eb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:19.251401 1356 tablet_service.cc:1467] Processing CreateTablet for tablet aaf1e6822ff14407b9f6270ef1515b29 (DEFAULT_TABLE table=test-table [id=6370281f312b414b9c069214f8e4eb4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:19.252238 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf1e6822ff14407b9f6270ef1515b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:19.252300 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf1e6822ff14407b9f6270ef1515b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:19.257457 1441 tablet_bootstrap.cc:492] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: Bootstrap starting.
I20260927 12:35:19.260730 1442 tablet_bootstrap.cc:492] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: Bootstrap starting.
I20260927 12:35:19.262413 1441 tablet_bootstrap.cc:654] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:19.263348 1441 log.cc:824] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:19.263805 1442 tablet_bootstrap.cc:654] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:19.264232 1443 tablet_bootstrap.cc:492] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: Bootstrap starting.
I20260927 12:35:19.265158 1442 log.cc:824] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:19.266105 1441 tablet_bootstrap.cc:492] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: No bootstrap required, opened a new log
I20260927 12:35:19.266311 1441 ts_tablet_manager.cc:1397] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 12:35:19.266497 1443 tablet_bootstrap.cc:654] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:19.267295 1442 tablet_bootstrap.cc:492] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: No bootstrap required, opened a new log
I20260927 12:35:19.267336 1443 log.cc:824] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:19.267515 1442 ts_tablet_manager.cc:1397] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 12:35:19.269567 1443 tablet_bootstrap.cc:492] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: No bootstrap required, opened a new log
I20260927 12:35:19.269819 1443 ts_tablet_manager.cc:1397] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:35:19.270591 1441 raft_consensus.cc:348] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.270931 1441 raft_consensus.cc:374] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:19.270996 1441 raft_consensus.cc:729] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdf2e0508334841bc9aefedd1dc49ea, State: Initialized, Role: FOLLOWER
I20260927 12:35:19.270912 1442 raft_consensus.cc:348] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.271176 1442 raft_consensus.cc:374] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:19.271231 1442 raft_consensus.cc:729] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de9b4c531e148c3b2b68f576c18092d, State: Initialized, Role: FOLLOWER
I20260927 12:35:19.271543 1441 consensus_queue.cc:260] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.271622 1442 consensus_queue.cc:260] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.272245 1422 heartbeater.cc:503] Master 127.0.98.62:45789 was elected leader, sending a full tablet report...
I20260927 12:35:19.272624 1442 ts_tablet_manager.cc:1428] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:35:19.272637 1441 ts_tablet_manager.cc:1428] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:35:19.274403 1443 raft_consensus.cc:348] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.274626 1443 raft_consensus.cc:374] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:19.274672 1443 raft_consensus.cc:729] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85c3ddd9cbf415f9d2012ac0af47653, State: Initialized, Role: FOLLOWER
I20260927 12:35:19.275110 1443 consensus_queue.cc:260] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.276371 1443 ts_tablet_manager.cc:1428] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 12:35:19.438215 1423 tablet.cc:2369] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:19.465518 1154 tablet.cc:2369] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:19.480893 1449 raft_consensus.cc:482] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:19.481163 1449 raft_consensus.cc:504] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.482414 1449 leader_election.cc:288] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcdf2e0508334841bc9aefedd1dc49ea (127.0.98.2:45415), 7de9b4c531e148c3b2b68f576c18092d (127.0.98.3:44103)
I20260927 12:35:19.486195 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.482761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32869 (local address 127.0.98.3:44103)
0927 12:35:19.482896 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.482901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.483100 (+ 199us) server_negotiation.cc:407] Connection header received
0927 12:35:19.483235 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.483237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.483288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.483350 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:19.483582 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.484123 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.485238 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.485669 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.485767 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.485833 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.485987 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.486059 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:35:19.486225 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.482665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44103 (local address 127.0.98.1:32869)
0927 12:35:19.482998 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:19.483026 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:19.483150 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:19.483368 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:19.483370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:19.483385 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:19.483471 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.483475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.484216 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.484220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.485111 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.485121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.485791 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.485795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.485879 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.485898 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:35:19.485964 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:35:19.486055 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":25,"spinlock_wait_cycles":22656,"thread_start_us":132,"threads_started":1}
I20260927 12:35:19.486370 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.482813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52327 (local address 127.0.98.2:45415)
0927 12:35:19.483011 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.483014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.483131 (+ 117us) server_negotiation.cc:407] Connection header received
0927 12:35:19.483455 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.483458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.483497 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.483556 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:19.483796 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.484347 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.485431 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.485860 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.485955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.486016 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.486192 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.486268 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 12:35:19.486387 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.482673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45415 (local address 127.0.98.1:52327)
0927 12:35:19.482999 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:19.483013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:19.483150 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:19.483592 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:19.483595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:19.483613 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:19.483698 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.483703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.484446 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.484450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.485313 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:19.485322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.485988 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.485991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:19.486083 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.486100 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:35:19.486163 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:35:19.486233 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":30,"spinlock_wait_cycles":9344,"thread_start_us":100,"threads_started":1}
I20260927 12:35:19.487260 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf1e6822ff14407b9f6270ef1515b29" candidate_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de9b4c531e148c3b2b68f576c18092d" is_pre_election: true
I20260927 12:35:19.487258 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf1e6822ff14407b9f6270ef1515b29" candidate_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" is_pre_election: true
I20260927 12:35:19.487583 1241 raft_consensus.cc:2454] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85c3ddd9cbf415f9d2012ac0af47653 in term 0.
I20260927 12:35:19.487583 1376 raft_consensus.cc:2454] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c85c3ddd9cbf415f9d2012ac0af47653 in term 0.
I20260927 12:35:19.488121 1043 leader_election.cc:302] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7de9b4c531e148c3b2b68f576c18092d, c85c3ddd9cbf415f9d2012ac0af47653, dcdf2e0508334841bc9aefedd1dc49ea; no voters:
I20260927 12:35:19.488391 1449 raft_consensus.cc:2781] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:19.488471 1449 raft_consensus.cc:482] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:19.488500 1449 raft_consensus.cc:3037] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:35:19.517771 1288 tablet.cc:2369] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:19.531523 1449 raft_consensus.cc:504] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.532148 1449 leader_election.cc:288] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [CANDIDATE]: Term 1 election: Requested vote from peers dcdf2e0508334841bc9aefedd1dc49ea (127.0.98.2:45415), 7de9b4c531e148c3b2b68f576c18092d (127.0.98.3:44103)
I20260927 12:35:19.532644 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf1e6822ff14407b9f6270ef1515b29" candidate_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdf2e0508334841bc9aefedd1dc49ea"
I20260927 12:35:19.532693 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf1e6822ff14407b9f6270ef1515b29" candidate_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de9b4c531e148c3b2b68f576c18092d"
I20260927 12:35:19.532811 1376 raft_consensus.cc:3037] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:19.532812 1241 raft_consensus.cc:3037] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:19.534276 1376 raft_consensus.cc:2454] T aaf1e6822ff14407b9f6270ef1515b29 P 7de9b4c531e148c3b2b68f576c18092d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85c3ddd9cbf415f9d2012ac0af47653 in term 1.
I20260927 12:35:19.534318 1241 raft_consensus.cc:2454] T aaf1e6822ff14407b9f6270ef1515b29 P dcdf2e0508334841bc9aefedd1dc49ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c85c3ddd9cbf415f9d2012ac0af47653 in term 1.
I20260927 12:35:19.534745 1043 leader_election.cc:302] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7de9b4c531e148c3b2b68f576c18092d, c85c3ddd9cbf415f9d2012ac0af47653, dcdf2e0508334841bc9aefedd1dc49ea; no voters:
I20260927 12:35:19.534960 1449 raft_consensus.cc:2781] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:19.535198 1449 raft_consensus.cc:686] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [term 1 LEADER]: Becoming Leader. State: Replica: c85c3ddd9cbf415f9d2012ac0af47653, State: Running, Role: LEADER
I20260927 12:35:19.535636 1449 consensus_queue.cc:237] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } }
I20260927 12:35:19.539134 962 catalog_manager.cc:5495] T aaf1e6822ff14407b9f6270ef1515b29 P c85c3ddd9cbf415f9d2012ac0af47653 reported cstate change: term changed from 0 to 1, leader changed from <none> to c85c3ddd9cbf415f9d2012ac0af47653 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdf2e0508334841bc9aefedd1dc49ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7de9b4c531e148c3b2b68f576c18092d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c85c3ddd9cbf415f9d2012ac0af47653" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41517 } health_report { overall_health: HEALTHY } } }
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:19.622934 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.610986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.0.98.1:41517)
0927 12:35:19.611127 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.611132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.611459 (+ 327us) server_negotiation.cc:407] Connection header received
0927 12:35:19.611590 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.611593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.611645 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.611716 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:19.612512 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.613113 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.614042 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.614341 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.614459 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.618142 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:19.618157 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:19.618173 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:19.618205 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:19.620197 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.620739 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.620748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.620749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.620949 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.621241 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.621245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.621247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.621970 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:19.622256 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.622637 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.622771 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:19.718907 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.709414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.0.98.1:41517)
0927 12:35:19.709534 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.709539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.709732 (+ 193us) server_negotiation.cc:407] Connection header received
0927 12:35:19.709871 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.709874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.709920 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.709987 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:19.710707 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.711287 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.712240 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.712500 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.712592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.715028 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:19.715053 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:19.715055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:19.715082 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:19.716862 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.717339 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.717343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.717344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.717427 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.717705 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.717709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.717710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.718051 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:19.718202 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.718529 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.718665 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:35:19 dist-test-slave-tzgx krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:19.815014 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:19.805894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.0.98.1:41517)
0927 12:35:19.806022 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:19.806027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:19.806194 (+ 167us) server_negotiation.cc:407] Connection header received
0927 12:35:19.806382 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:19.806384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:19.806430 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:19.806497 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:19.807285 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.807823 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.808826 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:19.809185 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:19.809317 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:19.811992 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:19.811996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:19.811998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:19.812029 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:19.813376 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.813839 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.813843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.813844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.813923 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:19.814199 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:19.814203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:19.814204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:19.814390 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:19.814506 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:19.814739 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 12:35:19.814856 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 12:35:19.816416 1067 tablet_service.cc:2908] Rejecting scan request for tablet aaf1e6822ff14407b9f6270ef1515b29: Uninitialized: clean time has not yet been initialized
I20260927 12:35:19.818464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1024
I20260927 12:35:19.823719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1158
I20260927 12:35:19.828917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1293
I20260927 12:35:19.833895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 930
2026-09-27T12:35:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4931 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:20 dist-test-slave-tzgx krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:20 dist-test-slave-tzgx krb5kdc[1466](info): set up 2 sockets
Sep 27 12:35:20 dist-test-slave-tzgx krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260927 12:35:21.840705 392 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 12:35:22.023932 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.183s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:22.127631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:22.237807 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:22.238122 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:22.238180 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:22.242597 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:22.242691 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:35:22.242717 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:22.242739 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:22.242762 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:22.242792 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:22.249343 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34319
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:22.250560 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:22.251644 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:22.261025 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:22.261046 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:22.261025 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:22.261322 1479 server_base.cc:1034] running on GCE node
I20260927 12:35:22.262158 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:22.263402 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:22.264577 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790512522264552 us; error 43 us; skew 500 ppm
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:22.267395 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:22.268419 1479 webserver.cc:460] Webserver started at http://127.0.98.62:39029/ using document root <none> and password file <none>
I20260927 12:35:22.268989 1479 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:22.269060 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:22.269268 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:22.271481 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "3ffdffa91b4d4c31a5dc94b8b41be125"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "840d2fa452cb8d344a8c462e6938cab1"
server_key_iv: "dad9047960c060e1fbb5a6425e3b16ed"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.272069 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ffdffa91b4d4c31a5dc94b8b41be125"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "840d2fa452cb8d344a8c462e6938cab1"
server_key_iv: "dad9047960c060e1fbb5a6425e3b16ed"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.276125 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:35:22.278779 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:22.280134 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:22.280259 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "3ffdffa91b4d4c31a5dc94b8b41be125"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "840d2fa452cb8d344a8c462e6938cab1"
server_key_iv: "dad9047960c060e1fbb5a6425e3b16ed"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.280372 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:22.315413 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:22.318838 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:22.319038 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:22.326902 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34319
I20260927 12:35:22.326982 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34319 every 8 connection(s)
I20260927 12:35:22.327884 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:22.330756 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:22.334514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1479
I20260927 12:35:22.334626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:22.334887 392 external_mini_cluster.cc:1428] Setting key ae27058e78e1a71e60a66c044312e09b
I20260927 12:35:22.336843 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: Bootstrap starting.
I20260927 12:35:22.339069 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:22.339797 1557 log.cc:824] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:22.342120 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: No bootstrap required, opened a new log
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:22.346385 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 12:35:22.346695 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:22.346768 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffdffa91b4d4c31a5dc94b8b41be125, State: Initialized, Role: FOLLOWER
I20260927 12:35:22.347270 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 12:35:22.347416 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:22.347494 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:22.347594 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:22.348563 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 12:35:22.348935 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ffdffa91b4d4c31a5dc94b8b41be125; no voters:
I20260927 12:35:22.349298 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:22.349382 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:22.349606 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [term 1 LEADER]: Becoming Leader. State: Replica: 3ffdffa91b4d4c31a5dc94b8b41be125, State: Running, Role: LEADER
I20260927 12:35:22.349967 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260927 12:35:22.350276 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.336399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.0.98.62:34319)
0927 12:35:22.336880 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:22.336896 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:22.336942 (+ 46us) server_negotiation.cc:407] Connection header received
0927 12:35:22.337632 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:22.337650 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:22.337988 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:22.338319 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:22.339069 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.340168 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.341467 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.341976 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.342114 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.344888 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:22.344895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:22.344915 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:22.344958 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:22.346985 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.347463 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.347470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.347477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.347627 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.347936 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.347939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.347941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.348427 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:22.348616 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:22.348902 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 12:35:22.349231 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":143,"threads_started":1}
I20260927 12:35:22.350615 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:22.351696 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ffdffa91b4d4c31a5dc94b8b41be125. Latest consensus state: current_term: 1 leader_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 12:35:22.351807 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:22.351938 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffdffa91b4d4c31a5dc94b8b41be125" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260927 12:35:22.352005 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:22.352701 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:22.356007 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:22.362088 1570 catalog_manager.cc:1347] Generated new cluster ID: 1ecb17ddfbe0450290b4cc94cd93b287
I20260927 12:35:22.362193 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:22.390529 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:22.391618 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:22.399087 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: Generated new TSK 0
I20260927 12:35:22.399853 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:22.435643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:22.549311 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:22.549638 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:22.549711 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:22.554136 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:22.554217 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:22.554277 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:22.554342 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:22.559857 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:22.561077 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:22.562145 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:22.572608 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:22.572610 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:22.572808 1583 server_base.cc:1034] running on GCE node
W20260927 12:35:22.572610 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:22.573367 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:22.574059 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:22.575212 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790512522575187 us; error 39 us; skew 500 ppm
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:22.578253 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:22.579319 1583 webserver.cc:460] Webserver started at http://127.0.98.1:32925/ using document root <none> and password file <none>
I20260927 12:35:22.579888 1583 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:22.579960 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:22.580209 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:22.582402 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "48a2d324c14c4eacaea977c92e596ed1"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "d938e2db80fa166b468343c3fea60d7d"
server_key_iv: "cc65c531018df16814ac0d3c4a12f7fc"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.582994 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48a2d324c14c4eacaea977c92e596ed1"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "d938e2db80fa166b468343c3fea60d7d"
server_key_iv: "cc65c531018df16814ac0d3c4a12f7fc"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.587119 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 12:35:22.590647 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:22.592199 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:22.592350 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "48a2d324c14c4eacaea977c92e596ed1"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "d938e2db80fa166b468343c3fea60d7d"
server_key_iv: "cc65c531018df16814ac0d3c4a12f7fc"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.592492 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:22.609156 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:22.612587 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:22.612782 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:22.614331 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:22.614394 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:22.614450 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:22.614478 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:22.624722 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38003
I20260927 12:35:22.624850 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38003 every 8 connection(s)
I20260927 12:35:22.625696 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:22.628594 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.615146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55927 (local address 127.0.98.62:34319)
0927 12:35:22.615321 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:22.615325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:22.616257 (+ 932us) server_negotiation.cc:407] Connection header received
0927 12:35:22.617306 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:22.617310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:22.617365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:22.617451 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:22.619221 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.619808 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.621564 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.621869 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.621956 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.625093 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:22.625097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:22.625100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:22.625131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:22.626648 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.627191 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.627195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.627196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.627275 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.627572 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.627574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.627576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.627755 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:22.627833 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:22.628367 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 12:35:22.628473 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 12:35:22.629400 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.615583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:55927)
0927 12:35:22.616061 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:22.616143 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:22.617090 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:22.617671 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:22.617736 (+ 65us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:22.618288 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:22.619038 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.619061 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.619928 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.619931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.621427 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.621440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.622020 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.622025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.622237 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.622896 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:22.622919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:22.624887 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:22.626775 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.626781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.626793 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.627081 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.627345 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.627347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.627349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.627481 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.627831 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:22.627836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:22.628129 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 12:35:22.628314 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 12:35:22.628578 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
I20260927 12:35:22.630718 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 12:35:22.630869 1714 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:22.631366 1714 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 12:35:22.632731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1583
I20260927 12:35:22.632845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:22.633070 1510 ts_manager.cc:194] Registered new tserver with Master: 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003)
I20260927 12:35:22.633139 392 external_mini_cluster.cc:1428] Setting key f312c8f1aad03c416ca969e9d48c2757
I20260927 12:35:22.634460 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55927
I20260927 12:35:22.642366 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.633515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58951 (local address 127.0.98.62:34319)
0927 12:35:22.635032 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:22.635037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:22.635053 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:22.635092 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:22.635095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:22.635138 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:22.635233 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:22.635763 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.636527 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.637479 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.637694 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.637782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.639132 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:22.639135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:22.639137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:22.639161 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:22.640544 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.641075 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.641078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.641079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.641158 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.641498 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.641501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.641502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.641700 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:22.641786 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:22.642131 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 12:35:22.642239 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452}
I20260927 12:35:22.642546 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.632966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:58951)
0927 12:35:22.633349 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:22.633366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:22.633461 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:22.635264 (+ 1803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:22.635267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:22.635486 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:22.635631 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.635639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.636699 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.636702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.637346 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.637356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.637827 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.637832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.637927 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.638444 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:22.638462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:22.638969 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:22.640666 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.640670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.640673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.640965 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.641254 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.641258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.641260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.641377 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.641801 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:22.641805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:22.641926 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 12:35:22.642054 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 12:35:22.642306 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:22.664160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:22.776638 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:22.776935 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:22.776981 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:22.781168 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:22.781234 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:22.781275 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:22.781318 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:22.786811 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:22.787870 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:22.789036 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:22.798787 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:22.798955 1723 server_base.cc:1034] running on GCE node
W20260927 12:35:22.798791 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:22.799137 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:22.799563 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:22.800333 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:22.801522 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790512522801487 us; error 47 us; skew 500 ppm
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:22.804641 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:22.805637 1723 webserver.cc:460] Webserver started at http://127.0.98.2:46731/ using document root <none> and password file <none>
I20260927 12:35:22.806257 1723 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:22.806336 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:22.806552 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:22.808866 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "312bd610e92146c19e5e7672adad6865"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "3245dd95c0ea0f55447539c784ddfeca"
server_key_iv: "c82bb70555a3deadfe5c53ec0d3f44e6"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.809482 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "312bd610e92146c19e5e7672adad6865"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "3245dd95c0ea0f55447539c784ddfeca"
server_key_iv: "c82bb70555a3deadfe5c53ec0d3f44e6"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.813551 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:22.816237 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:22.817520 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:35:22.817647 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "312bd610e92146c19e5e7672adad6865"
format_stamp: "Formatted at 2026-09-27 12:35:22 on dist-test-slave-tzgx"
server_key: "3245dd95c0ea0f55447539c784ddfeca"
server_key_iv: "c82bb70555a3deadfe5c53ec0d3f44e6"
server_key_version: "encryptionkey@0"
I20260927 12:35:22.817754 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:22.831872 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:22.834898 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:22.835084 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:22.836575 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:22.836625 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:22.836676 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:22.836702 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:22 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:22.847033 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43467
I20260927 12:35:22.847057 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43467 every 8 connection(s)
I20260927 12:35:22.848174 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:22.850389 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.837349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49849 (local address 127.0.98.62:34319)
0927 12:35:22.837460 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:22.837463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:22.838606 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 12:35:22.839712 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:22.839717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:22.839779 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:22.839837 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:22.841555 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.842144 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.843411 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.843733 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.843846 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.846732 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:22.846736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:22.846739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:22.846773 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:22.848318 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.848884 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.848888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.848890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.848986 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.849320 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.849323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.849325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.849517 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:22.849603 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:22.850101 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 12:35:22.850230 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:35:22.850744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1723
I20260927 12:35:22.850841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:22.851137 392 external_mini_cluster.cc:1428] Setting key 186ff7bfeac0257f6e5f13edaef7d4e0
I20260927 12:35:22.851454 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.837934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:49849)
0927 12:35:22.838434 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:22.838479 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:22.839435 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:22.840305 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:22.840316 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:22.840818 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:22.841393 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.841414 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.842268 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.842271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.843298 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.843308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.843909 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.843914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.844175 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.844793 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:22.844815 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:22.846543 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:22.848428 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.848434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.848445 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.848748 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.849092 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.849095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.849097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.849230 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.849615 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:22.849622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:22.849865 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 12:35:22.850041 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 12:35:22.850361 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260927 12:35:22.852758 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 12:35:22.852910 1854 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:22.853461 1854 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 12:35:22.854578 1510 ts_manager.cc:194] Registered new tserver with Master: 312bd610e92146c19e5e7672adad6865 (127.0.98.2:43467)
I20260927 12:35:22.855337 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49849
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:22.862221 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.855108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48937 (local address 127.0.98.62:34319)
0927 12:35:22.855346 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:22.855350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:22.855396 (+ 46us) server_negotiation.cc:407] Connection header received
0927 12:35:22.855528 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:22.855531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:22.855587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:22.855652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:22.856279 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.856934 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.857745 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.858074 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.858162 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.859238 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:22.859242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:22.859245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:22.859271 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:22.860619 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.860997 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.861000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.861002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.861081 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:22.861323 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:22.861326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:22.861328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:22.861608 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:22.861705 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:22.861960 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 12:35:22.862068 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260927 12:35:22.862278 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:22.855062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:48937)
0927 12:35:22.855327 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:22.855343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:22.855444 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:22.855786 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:22.855789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:22.856013 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:22.856168 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.856177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.857045 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.857048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.857633 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:22.857641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:22.858181 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:22.858185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:22.858282 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:22.858761 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:22.858775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:22.859107 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:22.860715 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.860718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.860721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.860913 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.861159 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:22.861162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:22.861164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:22.861246 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:22.861704 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:22.861706 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:22.861802 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 12:35:22.861888 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 12:35:22.862045 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:22.878567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:22.990974 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:22.991269 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:22.991329 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:22.995429 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:22.995493 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:22.995549 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:22.995626 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:23.001157 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:23.002449 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:23.003625 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:35:23.013397 1863 server_base.cc:1034] running on GCE node
W20260927 12:35:23.013337 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:23.013342 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:23.013342 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:23.014024 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:23.014659 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:23.015841 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790512523015808 us; error 43 us; skew 500 ppm
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:23.018589 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:23.019541 1863 webserver.cc:460] Webserver started at http://127.0.98.3:45369/ using document root <none> and password file <none>
I20260927 12:35:23.020128 1863 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:23.020203 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:23.020409 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:23.022567 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "119297e7373c403296608d3913538cd4"
format_stamp: "Formatted at 2026-09-27 12:35:23 on dist-test-slave-tzgx"
server_key: "aff67e3f0d002ae62dcc01b1de2f7e3c"
server_key_iv: "5bd98ae763c3bd056719f389666e446f"
server_key_version: "encryptionkey@0"
I20260927 12:35:23.023111 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "119297e7373c403296608d3913538cd4"
format_stamp: "Formatted at 2026-09-27 12:35:23 on dist-test-slave-tzgx"
server_key: "aff67e3f0d002ae62dcc01b1de2f7e3c"
server_key_iv: "5bd98ae763c3bd056719f389666e446f"
server_key_version: "encryptionkey@0"
I20260927 12:35:23.027292 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:35:23.030428 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:23.031913 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 12:35:23.032042 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "119297e7373c403296608d3913538cd4"
format_stamp: "Formatted at 2026-09-27 12:35:23 on dist-test-slave-tzgx"
server_key: "aff67e3f0d002ae62dcc01b1de2f7e3c"
server_key_iv: "5bd98ae763c3bd056719f389666e446f"
server_key_version: "encryptionkey@0"
I20260927 12:35:23.032196 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:23.049881 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:23.053395 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:23.053584 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:23.055131 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:23.055184 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:23.055239 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:23.055267 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:23.065265 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33347
I20260927 12:35:23.065433 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33347 every 8 connection(s)
I20260927 12:35:23.066385 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:23.068645 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.055951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58369 (local address 127.0.98.62:34319)
0927 12:35:23.056116 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.056121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.057047 (+ 926us) server_negotiation.cc:407] Connection header received
0927 12:35:23.058098 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.058102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.058160 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.058231 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:23.059473 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.060013 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.061176 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.061415 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.061516 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.064759 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:23.064764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:23.064767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:23.064800 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:23.066453 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.067078 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.067082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.067084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.067181 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.067507 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.067510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.067513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.067704 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:23.067785 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.068345 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.068472 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:35:23.069422 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.056429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:58369)
0927 12:35:23.056878 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.056931 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.057860 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.058428 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.058435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.058765 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:23.059324 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.059339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.060159 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.060164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.061056 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.061066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.061707 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.061712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.061885 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.062448 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:23.062467 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:23.064541 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:23.066600 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:23.066606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:23.066621 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:23.066953 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:23.067269 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:23.067273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:23.067275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:23.067404 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:23.067808 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:23.067814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:23.068062 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.068274 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.068605 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260927 12:35:23.070639 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260927 12:35:23.070781 1994 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:23.071277 1994 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260927 12:35:23.072546 1509 ts_manager.cc:194] Registered new tserver with Master: 119297e7373c403296608d3913538cd4 (127.0.98.3:33347)
I20260927 12:35:23.073246 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58369
I20260927 12:35:23.075183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 1863
I20260927 12:35:23.075294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:23.075587 392 external_mini_cluster.cc:1428] Setting key 85dc5415272a00cc07e62b9bf4055416
I20260927 12:35:23.077728 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:35:23.080811 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.072865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52805 (local address 127.0.98.62:34319)
0927 12:35:23.072989 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.072994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.073730 (+ 736us) server_negotiation.cc:407] Connection header received
0927 12:35:23.073851 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.073854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.073902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.073977 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:23.074448 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.075044 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.075956 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.076214 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.076308 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.077552 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:23.077556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:23.077559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:23.077586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:23.079224 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.079636 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.079640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.079642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.079719 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.079958 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.079961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.079963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.080193 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:23.080281 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.080556 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.080658 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:35:23.080825 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.072778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:52805)
0927 12:35:23.073638 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.073651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.073757 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.074002 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.074006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.074215 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:23.074332 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.074340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.075188 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.075191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.075843 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.075853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.076343 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.076348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.076455 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.077038 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:23.077054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:23.077415 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:23.079333 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:23.079336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:23.079338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:23.079538 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:23.079798 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:23.079801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:23.079804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:23.079879 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:23.080279 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:23.080282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:23.080384 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.080483 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.080640 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":106,"threads_started":1}
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512523, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512523, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:23.103600 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.095198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.0.98.62:34319)
0927 12:35:23.095352 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.095355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.095367 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:35:23.095430 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.095433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.095473 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.095547 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:23.096393 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.097008 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.097847 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.098063 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.098154 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.100515 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:23.100518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:23.100520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:23.100548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:23.101968 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.102418 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.102421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.102422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.102512 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.102768 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.102772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.102773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.102960 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:23.103073 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.103300 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.103412 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:35:23.106689 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:23.109789 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:23.125005 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.119198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.98.3:33347)
0927 12:35:23.119749 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.119757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.119774 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:35:23.119831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.119835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.119989 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.120148 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.120623 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.121860 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.123422 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.124250 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.124365 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.124547 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.124734 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.124837 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":110,"threads_started":1}
I20260927 12:35:23.125036 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.118647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43467 (local address 127.0.0.1:33920)
0927 12:35:23.118924 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.118940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.119042 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.120257 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.120264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.120286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.120493 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.120500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.122003 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.122006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.123477 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.123489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.124519 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.124523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.124620 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.124643 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.124691 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.124829 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":60,"spinlock_wait_cycles":4992,"thread_start_us":108,"threads_started":1}
I20260927 12:35:23.125350 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.119165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.0.98.2:43467)
0927 12:35:23.119666 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.119681 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.119707 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:35:23.119788 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.119794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.120017 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.120250 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.120623 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.121860 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.123610 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.124383 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.124946 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.125104 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.125153 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.125239 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":115,"threads_started":1}
I20260927 12:35:23.125514 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.119060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33347 (local address 127.0.0.1:46106)
0927 12:35:23.119272 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.119286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.119368 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.120292 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.120294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.120315 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.120493 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.120500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.121989 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.121993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.123307 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.123316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.124386 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.124391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.124510 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.124560 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.124678 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.124861 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":190,"spinlock_wait_cycles":2304,"thread_start_us":89,"threads_started":1}
I20260927 12:35:23.125717 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.118487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.0.98.1:38003)
0927 12:35:23.120300 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.120318 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.120342 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:35:23.120420 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.120426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.120616 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.120762 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.120978 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.121853 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.123765 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.124448 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.124557 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.124724 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.125468 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.125572 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":90,"threads_started":1}
I20260927 12:35:23.125967 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.118326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38003 (local address 127.0.0.1:35198)
0927 12:35:23.118679 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.118729 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.118849 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.120752 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.120756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.120775 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.120878 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.120886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.121968 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.121971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.123649 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.123657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.124786 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.124790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.124890 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.124914 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.124987 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.125051 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":328,"thread_start_us":87,"threads_started":1}
I20260927 12:35:23.128059 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 7540884e01514847ad77a4e0e1036cd1 (DEFAULT_TABLE table=test-table [id=30b4ced4d8ad4121866321c3c6bcfb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:23.128587 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 7540884e01514847ad77a4e0e1036cd1 (DEFAULT_TABLE table=test-table [id=30b4ced4d8ad4121866321c3c6bcfb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:23.129200 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7540884e01514847ad77a4e0e1036cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.129503 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7540884e01514847ad77a4e0e1036cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.130772 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 7540884e01514847ad77a4e0e1036cd1 (DEFAULT_TABLE table=test-table [id=30b4ced4d8ad4121866321c3c6bcfb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:23.131834 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7540884e01514847ad77a4e0e1036cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.136195 2017 tablet_bootstrap.cc:492] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Bootstrap starting.
I20260927 12:35:23.137848 2018 tablet_bootstrap.cc:492] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Bootstrap starting.
I20260927 12:35:23.138450 2017 tablet_bootstrap.cc:654] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.139123 2019 tablet_bootstrap.cc:492] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Bootstrap starting.
I20260927 12:35:23.139269 2017 log.cc:824] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:23.139986 2018 tablet_bootstrap.cc:654] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.140744 2018 log.cc:824] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:23.141796 2019 tablet_bootstrap.cc:654] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.141875 2017 tablet_bootstrap.cc:492] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: No bootstrap required, opened a new log
I20260927 12:35:23.142067 2017 ts_tablet_manager.cc:1397] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:35:23.142365 2018 tablet_bootstrap.cc:492] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: No bootstrap required, opened a new log
I20260927 12:35:23.142540 2018 ts_tablet_manager.cc:1397] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:35:23.142571 2019 log.cc:824] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:23.145370 2017 raft_consensus.cc:348] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.145457 2018 raft_consensus.cc:348] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.145654 2018 raft_consensus.cc:374] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.145658 2017 raft_consensus.cc:374] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.145706 2017 raft_consensus.cc:729] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119297e7373c403296608d3913538cd4, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.145713 2018 raft_consensus.cc:729] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312bd610e92146c19e5e7672adad6865, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.145812 2019 tablet_bootstrap.cc:492] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: No bootstrap required, opened a new log
I20260927 12:35:23.146118 2019 ts_tablet_manager.cc:1397] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:35:23.146153 2017 consensus_queue.cc:260] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.146347 2018 consensus_queue.cc:260] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.146878 1994 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 12:35:23.147178 2017 ts_tablet_manager.cc:1428] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:35:23.147182 2018 ts_tablet_manager.cc:1428] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:35:23.147503 1854 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 12:35:23.150938 2019 raft_consensus.cc:348] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.151268 2019 raft_consensus.cc:374] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.151330 2019 raft_consensus.cc:729] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a2d324c14c4eacaea977c92e596ed1, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.151901 2019 consensus_queue.cc:260] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.152603 1714 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260927 12:35:23.153093 2019 ts_tablet_manager.cc:1428] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:35:23.169556 2024 raft_consensus.cc:482] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:23.169780 2024 raft_consensus.cc:504] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.171030 2024 leader_election.cc:288] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003), 119297e7373c403296608d3913538cd4 (127.0.98.3:33347)
I20260927 12:35:23.174821 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.171296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38003 (local address 127.0.98.2:59737)
0927 12:35:23.171449 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.171465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.171561 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.171840 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.171843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.171865 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.172013 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.172019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.172900 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.172904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.173707 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.173717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.174361 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.174364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.174502 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.174529 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.174596 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.174678 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":22}
I20260927 12:35:23.174821 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.171356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59737 (local address 127.0.98.1:38003)
0927 12:35:23.171487 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.171492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.171508 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:23.171672 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.171676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.171725 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.171792 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.172165 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.172768 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.173835 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.174243 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.174339 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.174407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.174623 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.174705 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:35:23.175525 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7540884e01514847ad77a4e0e1036cd1" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a2d324c14c4eacaea977c92e596ed1" is_pre_election: true
I20260927 12:35:23.175719 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.171606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33347 (local address 127.0.98.2:37827)
0927 12:35:23.172041 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.172052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.172171 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.172390 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.172393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.172407 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.172528 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.172533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.173329 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.173333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.174542 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.174551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.175290 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.175294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.175388 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.175405 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.175468 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.175539 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":37,"thread_start_us":153,"threads_started":1}
I20260927 12:35:23.175714 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.171733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37827 (local address 127.0.98.3:33347)
0927 12:35:23.171848 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.171852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.172136 (+ 284us) server_negotiation.cc:407] Connection header received
0927 12:35:23.172263 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.172266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.172311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.172368 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.172721 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.173222 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.174665 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.175172 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.175274 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.175334 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.175506 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.175595 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:35:23.175899 1668 raft_consensus.cc:2454] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 0.
I20260927 12:35:23.176484 1742 leader_election.cc:302] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 312bd610e92146c19e5e7672adad6865, 48a2d324c14c4eacaea977c92e596ed1; no voters:
I20260927 12:35:23.176751 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7540884e01514847ad77a4e0e1036cd1" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119297e7373c403296608d3913538cd4" is_pre_election: true
I20260927 12:35:23.176781 2024 raft_consensus.cc:2781] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:23.176857 2024 raft_consensus.cc:482] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:23.176887 2024 raft_consensus.cc:3037] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.177038 1948 raft_consensus.cc:2454] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 0.
I20260927 12:35:23.178077 2024 raft_consensus.cc:504] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.178423 2024 leader_election.cc:288] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 election: Requested vote from peers 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003), 119297e7373c403296608d3913538cd4 (127.0.98.3:33347)
I20260927 12:35:23.178799 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7540884e01514847ad77a4e0e1036cd1" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119297e7373c403296608d3913538cd4"
I20260927 12:35:23.178800 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7540884e01514847ad77a4e0e1036cd1" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a2d324c14c4eacaea977c92e596ed1"
I20260927 12:35:23.178918 1668 raft_consensus.cc:3037] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.178915 1948 raft_consensus.cc:3037] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.179965 1668 raft_consensus.cc:2454] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 1.
I20260927 12:35:23.180130 1948 raft_consensus.cc:2454] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 1.
I20260927 12:35:23.180233 1742 leader_election.cc:302] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 312bd610e92146c19e5e7672adad6865, 48a2d324c14c4eacaea977c92e596ed1; no voters:
I20260927 12:35:23.180405 2024 raft_consensus.cc:2781] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:23.180666 2024 raft_consensus.cc:686] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 1 LEADER]: Becoming Leader. State: Replica: 312bd610e92146c19e5e7672adad6865, State: Running, Role: LEADER
I20260927 12:35:23.181044 2024 consensus_queue.cc:237] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } }
I20260927 12:35:23.184422 1510 catalog_manager.cc:5495] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 312bd610e92146c19e5e7672adad6865 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } health_report { overall_health: HEALTHY } } }
I20260927 12:35:23.208513 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.204706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.0.98.2:43467)
0927 12:35:23.204834 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.204838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.204859 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:35:23.205136 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.205138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.205196 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.205281 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:23.205550 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.206053 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.206947 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.207170 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.207259 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.207607 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.208059 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.208202 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.208295 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.208362 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:35:23.214442 1668 raft_consensus.cc:1261] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Refusing update from remote peer 312bd610e92146c19e5e7672adad6865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:23.214668 1948 raft_consensus.cc:1261] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Refusing update from remote peer 312bd610e92146c19e5e7672adad6865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:23.215121 2024 consensus_queue.cc:1035] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:23.215319 2027 consensus_queue.cc:1035] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:23.236610 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:23.236788 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:23.239265 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: Sending DeleteTablet for 3 replicas of tablet 7540884e01514847ad77a4e0e1036cd1
I20260927 12:35:23.240119 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7540884e01514847ad77a4e0e1036cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33920
I20260927 12:35:23.240176 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 7540884e01514847ad77a4e0e1036cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35198
I20260927 12:35:23.240715 2041 tablet_replica.cc:330] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: stopping tablet replica
I20260927 12:35:23.240829 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:35:23.240862 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 7540884e01514847ad77a4e0e1036cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46106
I20260927 12:35:23.241065 2041 raft_consensus.cc:2229] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 12:35:23.241236 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:23.241401 2041 raft_consensus.cc:2258] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:23.241812 2042 tablet_replica.cc:330] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: stopping tablet replica
I20260927 12:35:23.242195 2042 raft_consensus.cc:2229] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:23.242576 2042 raft_consensus.cc:2258] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:23.243034 2040 tablet_replica.cc:330] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: stopping tablet replica
I20260927 12:35:23.243119 2041 ts_tablet_manager.cc:1905] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:23.243404 2040 raft_consensus.cc:2229] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:23.243875 2040 raft_consensus.cc:2258] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:23.245520 2040 ts_tablet_manager.cc:1905] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:23.246049 2041 ts_tablet_manager.cc:1918] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:23.246168 2041 log.cc:1196] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/7540884e01514847ad77a4e0e1036cd1
I20260927 12:35:23.246487 2041 ts_tablet_manager.cc:1939] T 7540884e01514847ad77a4e0e1036cd1 P 48a2d324c14c4eacaea977c92e596ed1: Deleting consensus metadata
I20260927 12:35:23.247206 1788 tablet_service.cc:1467] Processing CreateTablet for tablet d84777f46a514c02bbbb4f216739f42a (DEFAULT_TABLE table=test-table [id=23d9d8bca8e84dc0bd4f2d16ed638fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:23.247226 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d84777f46a514c02bbbb4f216739f42a (DEFAULT_TABLE table=test-table [id=23d9d8bca8e84dc0bd4f2d16ed638fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:23.247480 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84777f46a514c02bbbb4f216739f42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.247480 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84777f46a514c02bbbb4f216739f42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.247830 1498 catalog_manager.cc:4841] TS 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003): tablet 7540884e01514847ad77a4e0e1036cd1 (table test-table [id=30b4ced4d8ad4121866321c3c6bcfb9c]) successfully deleted
I20260927 12:35:23.249543 2042 ts_tablet_manager.cc:1905] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:23.249828 2018 tablet_bootstrap.cc:492] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Bootstrap starting.
I20260927 12:35:23.250939 2018 tablet_bootstrap.cc:654] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.252430 2042 ts_tablet_manager.cc:1918] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:23.252539 2042 log.cc:1196] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/7540884e01514847ad77a4e0e1036cd1
I20260927 12:35:23.252754 2018 tablet_bootstrap.cc:492] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: No bootstrap required, opened a new log
I20260927 12:35:23.252840 2042 ts_tablet_manager.cc:1939] T 7540884e01514847ad77a4e0e1036cd1 P 119297e7373c403296608d3913538cd4: Deleting consensus metadata
I20260927 12:35:23.252851 2018 ts_tablet_manager.cc:1397] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:23.253342 2018 raft_consensus.cc:348] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.253454 2018 raft_consensus.cc:374] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.253495 2018 raft_consensus.cc:729] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312bd610e92146c19e5e7672adad6865, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.253625 2018 consensus_queue.cc:260] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.253863 2018 ts_tablet_manager.cc:1428] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:23.254580 1928 tablet_service.cc:1467] Processing CreateTablet for tablet d84777f46a514c02bbbb4f216739f42a (DEFAULT_TABLE table=test-table [id=23d9d8bca8e84dc0bd4f2d16ed638fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:23.255239 1498 catalog_manager.cc:4841] TS 119297e7373c403296608d3913538cd4 (127.0.98.3:33347): tablet 7540884e01514847ad77a4e0e1036cd1 (table test-table [id=30b4ced4d8ad4121866321c3c6bcfb9c]) successfully deleted
I20260927 12:35:23.255824 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84777f46a514c02bbbb4f216739f42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.258443 2017 tablet_bootstrap.cc:492] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Bootstrap starting.
I20260927 12:35:23.259101 2019 tablet_bootstrap.cc:492] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Bootstrap starting.
I20260927 12:35:23.259370 2017 tablet_bootstrap.cc:654] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.260061 2019 tablet_bootstrap.cc:654] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.261325 2040 ts_tablet_manager.cc:1918] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:23.261427 2019 tablet_bootstrap.cc:492] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: No bootstrap required, opened a new log
I20260927 12:35:23.261452 2040 log.cc:1196] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/7540884e01514847ad77a4e0e1036cd1
I20260927 12:35:23.261530 2019 ts_tablet_manager.cc:1397] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 12:35:23.261740 2040 ts_tablet_manager.cc:1939] T 7540884e01514847ad77a4e0e1036cd1 P 312bd610e92146c19e5e7672adad6865: Deleting consensus metadata
I20260927 12:35:23.261967 2019 raft_consensus.cc:348] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.262071 2019 raft_consensus.cc:374] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.262101 2019 raft_consensus.cc:729] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a2d324c14c4eacaea977c92e596ed1, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.262226 2019 consensus_queue.cc:260] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.262449 2019 ts_tablet_manager.cc:1428] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:35:23.263005 1497 catalog_manager.cc:4841] TS 312bd610e92146c19e5e7672adad6865 (127.0.98.2:43467): tablet 7540884e01514847ad77a4e0e1036cd1 (table test-table [id=30b4ced4d8ad4121866321c3c6bcfb9c]) successfully deleted
I20260927 12:35:23.263697 2017 tablet_bootstrap.cc:492] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: No bootstrap required, opened a new log
I20260927 12:35:23.263800 2017 ts_tablet_manager.cc:1397] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 12:35:23.264328 2017 raft_consensus.cc:348] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.264451 2017 raft_consensus.cc:374] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.264497 2017 raft_consensus.cc:729] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119297e7373c403296608d3913538cd4, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.264626 2017 consensus_queue.cc:260] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.264878 2017 ts_tablet_manager.cc:1428] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260927 12:35:23.317567 1995 tablet.cc:2369] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:23.349360 1855 tablet.cc:2369] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:23.377386 1715 tablet.cc:2369] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:23.395344 2023 raft_consensus.cc:482] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:23.395545 2023 raft_consensus.cc:504] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.396821 2023 leader_election.cc:288] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003), 312bd610e92146c19e5e7672adad6865 (127.0.98.2:43467)
I20260927 12:35:23.400555 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.397193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55795 (local address 127.0.98.1:38003)
0927 12:35:23.397342 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.397347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.397369 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:35:23.397460 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.397464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.397519 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.397613 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.397916 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.398416 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.399526 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.399939 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.400047 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.400148 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.400338 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.400424 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:35:23.400555 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.397104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38003 (local address 127.0.98.3:55795)
0927 12:35:23.397244 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.397260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.397358 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.397648 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.397652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.397671 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.397790 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.397797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.398528 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.398531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.399381 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.399391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.400065 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.400069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.400203 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.400223 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.400305 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.400398 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 12:35:23.400648 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.397450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38273 (local address 127.0.98.2:43467)
0927 12:35:23.397595 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.397600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.397613 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:23.397658 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.397660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.397706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.397764 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:23.398019 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.398495 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.399633 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.400082 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.400214 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.400281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.400460 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.400538 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:35:23.400722 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.397122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43467 (local address 127.0.98.3:38273)
0927 12:35:23.397455 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.397467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.397547 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.397782 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.397785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.397800 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:23.397916 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.397921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.398591 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.398594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.399485 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.399496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.400233 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.400236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.400337 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.400362 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.400425 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.400511 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260927 12:35:23.401135 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84777f46a514c02bbbb4f216739f42a" candidate_uuid: "119297e7373c403296608d3913538cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a2d324c14c4eacaea977c92e596ed1" is_pre_election: true
I20260927 12:35:23.401291 1668 raft_consensus.cc:2454] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119297e7373c403296608d3913538cd4 in term 0.
I20260927 12:35:23.401341 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84777f46a514c02bbbb4f216739f42a" candidate_uuid: "119297e7373c403296608d3913538cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312bd610e92146c19e5e7672adad6865" is_pre_election: true
I20260927 12:35:23.401612 1808 raft_consensus.cc:2454] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119297e7373c403296608d3913538cd4 in term 0.
I20260927 12:35:23.401682 1882 leader_election.cc:302] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119297e7373c403296608d3913538cd4, 48a2d324c14c4eacaea977c92e596ed1; no voters:
I20260927 12:35:23.401980 2023 raft_consensus.cc:2781] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:23.402045 2023 raft_consensus.cc:482] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:23.402089 2023 raft_consensus.cc:3037] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.403126 2023 raft_consensus.cc:504] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.403534 2023 leader_election.cc:288] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [CANDIDATE]: Term 1 election: Requested vote from peers 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003), 312bd610e92146c19e5e7672adad6865 (127.0.98.2:43467)
I20260927 12:35:23.404022 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84777f46a514c02bbbb4f216739f42a" candidate_uuid: "119297e7373c403296608d3913538cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312bd610e92146c19e5e7672adad6865"
I20260927 12:35:23.404057 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d84777f46a514c02bbbb4f216739f42a" candidate_uuid: "119297e7373c403296608d3913538cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a2d324c14c4eacaea977c92e596ed1"
I20260927 12:35:23.404202 1808 raft_consensus.cc:3037] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.404210 1668 raft_consensus.cc:3037] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.405443 1808 raft_consensus.cc:2454] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119297e7373c403296608d3913538cd4 in term 1.
I20260927 12:35:23.405443 1668 raft_consensus.cc:2454] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119297e7373c403296608d3913538cd4 in term 1.
I20260927 12:35:23.405905 1882 leader_election.cc:302] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119297e7373c403296608d3913538cd4, 48a2d324c14c4eacaea977c92e596ed1; no voters:
I20260927 12:35:23.406127 2023 raft_consensus.cc:2781] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:23.406370 2023 raft_consensus.cc:686] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 119297e7373c403296608d3913538cd4, State: Running, Role: LEADER
I20260927 12:35:23.406636 2023 consensus_queue.cc:237] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.409192 1509 catalog_manager.cc:5495] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 119297e7373c403296608d3913538cd4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "119297e7373c403296608d3913538cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } health_report { overall_health: HEALTHY } } }
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:23.482290 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.473899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.0.1:54992)
0927 12:35:23.474223 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.474240 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.474345 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.474579 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.474581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.475017 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:23.475113 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.475119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.475922 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.475925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.476566 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.476575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.477071 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.477074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.477152 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.477794 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:23.477806 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:23.479253 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:23.480897 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:23.480905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:23.480911 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:23.481143 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:23.481339 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:23.481341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:23.481343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:23.481411 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:23.481798 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:23.481803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:23.481913 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.481999 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.482150 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":119,"threads_started":1}
I20260927 12:35:23.482324 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.473994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.0.98.62:34319)
0927 12:35:23.474135 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.474138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.474309 (+ 171us) server_negotiation.cc:407] Connection header received
0927 12:35:23.474421 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.474423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.474464 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.474551 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:23.475251 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.475802 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.476708 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.476939 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.477031 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.479411 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:23.479414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:23.479416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:23.479437 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:23.480807 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.481214 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.481216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.481217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.481280 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.481505 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.481508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.481510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.481713 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:23.481796 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.482076 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.482206 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":60}
I20260927 12:35:23.486086 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:35:23.486770 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:23.491262 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccd113585f846649868e15c7605e437 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c084de39a9a44569b72e56f3f0a202f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:23.491432 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccd113585f846649868e15c7605e437 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c084de39a9a44569b72e56f3f0a202f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:23.491433 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 1ccd113585f846649868e15c7605e437 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c084de39a9a44569b72e56f3f0a202f0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:23.491638 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccd113585f846649868e15c7605e437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.491791 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccd113585f846649868e15c7605e437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.491791 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ccd113585f846649868e15c7605e437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:23.494480 2019 tablet_bootstrap.cc:492] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: Bootstrap starting.
I20260927 12:35:23.495398 2019 tablet_bootstrap.cc:654] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.496047 2017 tablet_bootstrap.cc:492] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: Bootstrap starting.
I20260927 12:35:23.496428 2018 tablet_bootstrap.cc:492] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: Bootstrap starting.
I20260927 12:35:23.497002 2017 tablet_bootstrap.cc:654] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.497263 2018 tablet_bootstrap.cc:654] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:23.498320 2018 tablet_bootstrap.cc:492] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: No bootstrap required, opened a new log
I20260927 12:35:23.498320 2019 tablet_bootstrap.cc:492] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: No bootstrap required, opened a new log
I20260927 12:35:23.498415 2019 ts_tablet_manager.cc:1397] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 12:35:23.498415 2018 ts_tablet_manager.cc:1397] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:23.498839 2019 raft_consensus.cc:348] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.498839 2018 raft_consensus.cc:348] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.498968 2018 raft_consensus.cc:374] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.498976 2019 raft_consensus.cc:374] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.499001 2018 raft_consensus.cc:729] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312bd610e92146c19e5e7672adad6865, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.499018 2019 raft_consensus.cc:729] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a2d324c14c4eacaea977c92e596ed1, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.499140 2018 consensus_queue.cc:260] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.499140 2019 consensus_queue.cc:260] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.499414 2018 ts_tablet_manager.cc:1428] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:35:23.499414 2019 ts_tablet_manager.cc:1428] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:35:23.499425 2025 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.499538 2027 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.499651 2027 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.499645 2025 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.500157 2017 tablet_bootstrap.cc:492] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: No bootstrap required, opened a new log
I20260927 12:35:23.500254 2017 ts_tablet_manager.cc:1397] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 12:35:23.500722 2017 raft_consensus.cc:348] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.500869 2017 raft_consensus.cc:374] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:23.500909 2017 raft_consensus.cc:729] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119297e7373c403296608d3913538cd4, State: Initialized, Role: FOLLOWER
I20260927 12:35:23.501056 2017 consensus_queue.cc:260] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.501302 2023 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.501456 2023 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.501376 2017 ts_tablet_manager.cc:1428] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:35:23.601987 2027 raft_consensus.cc:482] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:23.602128 2027 raft_consensus.cc:504] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.602530 2027 leader_election.cc:288] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003), 119297e7373c403296608d3913538cd4 (127.0.98.3:33347)
I20260927 12:35:23.603060 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccd113585f846649868e15c7605e437" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119297e7373c403296608d3913538cd4" is_pre_election: true
I20260927 12:35:23.603076 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccd113585f846649868e15c7605e437" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a2d324c14c4eacaea977c92e596ed1" is_pre_election: true
I20260927 12:35:23.603191 1948 raft_consensus.cc:2454] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 0.
I20260927 12:35:23.603245 1668 raft_consensus.cc:2454] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 0.
I20260927 12:35:23.603505 1742 leader_election.cc:302] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119297e7373c403296608d3913538cd4, 312bd610e92146c19e5e7672adad6865; no voters:
I20260927 12:35:23.603680 2027 raft_consensus.cc:2781] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:23.603740 2027 raft_consensus.cc:482] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:23.603783 2027 raft_consensus.cc:3037] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.604851 2027 raft_consensus.cc:504] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.605199 2027 leader_election.cc:288] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 election: Requested vote from peers 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003), 119297e7373c403296608d3913538cd4 (127.0.98.3:33347)
I20260927 12:35:23.605696 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccd113585f846649868e15c7605e437" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119297e7373c403296608d3913538cd4"
I20260927 12:35:23.605816 1948 raft_consensus.cc:3037] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.605702 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ccd113585f846649868e15c7605e437" candidate_uuid: "312bd610e92146c19e5e7672adad6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48a2d324c14c4eacaea977c92e596ed1"
I20260927 12:35:23.605934 1668 raft_consensus.cc:3037] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:23.606775 1948 raft_consensus.cc:2454] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 1.
I20260927 12:35:23.606871 1668 raft_consensus.cc:2454] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 312bd610e92146c19e5e7672adad6865 in term 1.
I20260927 12:35:23.607111 1742 leader_election.cc:302] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119297e7373c403296608d3913538cd4, 312bd610e92146c19e5e7672adad6865; no voters:
I20260927 12:35:23.607312 2027 raft_consensus.cc:2781] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:23.607429 2027 raft_consensus.cc:686] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [term 1 LEADER]: Becoming Leader. State: Replica: 312bd610e92146c19e5e7672adad6865, State: Running, Role: LEADER
I20260927 12:35:23.607568 2027 consensus_queue.cc:237] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } }
I20260927 12:35:23.608274 2035 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: TxnStatusTablet state changed. Reason: New leader 312bd610e92146c19e5e7672adad6865. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.608397 2035 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:23.608625 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:35:23.608775 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:35:23.609417 1509 catalog_manager.cc:5495] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 312bd610e92146c19e5e7672adad6865 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:23.623628 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.619500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43467 (local address 127.0.0.1:33934)
0927 12:35:23.619644 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.619660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.619856 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.620280 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.620282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.620292 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:23.620377 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.620382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.621474 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.621477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.622265 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.622277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.622850 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.622853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.622957 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.623008 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.623337 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.623357 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.623424 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.623507 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 12:35:23.623647 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.619619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.0.98.2:43467)
0927 12:35:23.619749 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.619753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.619836 (+ 83us) server_negotiation.cc:407] Connection header received
0927 12:35:23.620024 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.620026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.620075 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.620165 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:23.620486 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.621324 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.622371 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.622717 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.622816 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.623144 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.623234 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.623301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.623442 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.623509 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:35:23.626689 1948 raft_consensus.cc:1261] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Refusing update from remote peer 312bd610e92146c19e5e7672adad6865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:23.626689 1668 raft_consensus.cc:1261] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Refusing update from remote peer 312bd610e92146c19e5e7672adad6865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:23.627164 2024 consensus_queue.cc:1035] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:23.627454 2024 consensus_queue.cc:1035] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:23.628666 2025 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: TxnStatusTablet state changed. Reason: New leader 312bd610e92146c19e5e7672adad6865. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.628780 2025 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.628724 2023 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: TxnStatusTablet state changed. Reason: New leader 312bd610e92146c19e5e7672adad6865. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.628815 2023 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.630370 2027 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.630492 2027 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:23.633984 2023 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.634122 2023 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 119297e7373c403296608d3913538cd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.634524 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333939296769077248 to 7333939296697970688. Current Snapshot: MvccSnapshot[applied={T|T < 7333939296769077248 or (T in {7333939296769077248})}]
I20260927 12:35:23.640323 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.636292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.0.98.3:33347)
0927 12:35:23.636421 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.636426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.636439 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:23.636514 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.636517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.636580 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.636704 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:23.637127 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.637615 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.638455 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.638748 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.638846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.639222 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.639828 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.639945 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.640112 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.640168 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:35:23.642259 2024 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.642395 2024 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 312bd610e92146c19e5e7672adad6865: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:23.643422 2025 tablet_replica.cc:439] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "312bd610e92146c19e5e7672adad6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 } } peers { permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 } } peers { permanent_uuid: "119297e7373c403296608d3913538cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33347 } } }
I20260927 12:35:23.643561 2025 tablet_replica.cc:442] T 1ccd113585f846649868e15c7605e437 P 48a2d324c14c4eacaea977c92e596ed1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:23.653962 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.650239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43467 (local address 127.0.98.3:54721)
0927 12:35:23.650661 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.650679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.650794 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.651012 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.651015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.651022 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:23.651156 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.651161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.651991 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.651995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.652758 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.652768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.653208 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.653211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.653307 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.653366 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.653661 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.653668 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.653727 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.653813 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":110,"threads_started":1}
I20260927 12:35:23.653947 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.650398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54721 (local address 127.0.98.2:43467)
0927 12:35:23.650538 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.650542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.650755 (+ 213us) server_negotiation.cc:407] Connection header received
0927 12:35:23.650880 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.650883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.650934 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.650993 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:23.651266 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.651882 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.652868 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.653108 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.653200 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.653501 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.653587 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.653647 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.653738 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.653794 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:35:23.661139 1668 raft_consensus.cc:1261] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Refusing update from remote peer 119297e7373c403296608d3913538cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:23.661186 1808 raft_consensus.cc:1261] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Refusing update from remote peer 119297e7373c403296608d3913538cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:23.661656 2047 consensus_queue.cc:1035] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a2d324c14c4eacaea977c92e596ed1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:23.661857 2023 consensus_queue.cc:1035] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "312bd610e92146c19e5e7672adad6865" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:23.665047 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333939296911085568 to 7333939295875940352. Current Snapshot: MvccSnapshot[applied={T|T < 7333939296911085568}]
I20260927 12:35:23.671160 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333939296911085568 to 7333939295875940352. Current Snapshot: MvccSnapshot[applied={T|T < 7333939296911085568}]
I20260927 12:35:23.683440 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.679232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44501 (local address 127.0.98.3:33347)
0927 12:35:23.679347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.679351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.679582 (+ 231us) server_negotiation.cc:407] Connection header received
0927 12:35:23.679709 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.679711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.679757 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.679816 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:23.680073 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.680686 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.681913 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.682313 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.682392 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.682831 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.682992 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.683073 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.683251 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.683317 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:35:23.683471 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.679155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33347 (local address 127.0.98.2:44501)
0927 12:35:23.679494 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:23.679509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:23.679624 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:23.679840 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:23.679843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:23.679850 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:23.679961 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.679967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.680791 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.680795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.681772 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:23.681783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.682501 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.682505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:23.682606 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.682662 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:23.683094 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:23.683101 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:35:23.683214 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 12:35:23.683309 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260927 12:35:23.708954 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:23.709134 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:23.711455 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ffdffa91b4d4c31a5dc94b8b41be125: Sending DeleteTablet for 3 replicas of tablet d84777f46a514c02bbbb4f216739f42a
I20260927 12:35:23.712347 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet d84777f46a514c02bbbb4f216739f42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35198
I20260927 12:35:23.712565 2041 tablet_replica.cc:330] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: stopping tablet replica
I20260927 12:35:23.712548 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet d84777f46a514c02bbbb4f216739f42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46106
I20260927 12:35:23.712683 2041 raft_consensus.cc:2229] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:23.712817 2041 raft_consensus.cc:2258] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:23.712800 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet d84777f46a514c02bbbb4f216739f42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33920
I20260927 12:35:23.712986 2040 tablet_replica.cc:330] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: stopping tablet replica
I20260927 12:35:23.712992 2042 tablet_replica.cc:330] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: stopping tablet replica
I20260927 12:35:23.713104 2042 raft_consensus.cc:2229] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:23.713105 2040 raft_consensus.cc:2229] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:23.713220 2040 raft_consensus.cc:2258] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:23.713436 2042 raft_consensus.cc:2258] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:23.714181 2040 ts_tablet_manager.cc:1905] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:23.715360 2041 ts_tablet_manager.cc:1905] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:23.715524 2042 ts_tablet_manager.cc:1905] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:23.717098 2040 ts_tablet_manager.cc:1918] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:23.717168 2040 log.cc:1196] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/d84777f46a514c02bbbb4f216739f42a
I20260927 12:35:23.717403 2040 ts_tablet_manager.cc:1939] T d84777f46a514c02bbbb4f216739f42a P 312bd610e92146c19e5e7672adad6865: Deleting consensus metadata
I20260927 12:35:23.718204 2041 ts_tablet_manager.cc:1918] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:23.718214 1497 catalog_manager.cc:4841] TS 312bd610e92146c19e5e7672adad6865 (127.0.98.2:43467): tablet d84777f46a514c02bbbb4f216739f42a (table test-table [id=23d9d8bca8e84dc0bd4f2d16ed638fc2]) successfully deleted
I20260927 12:35:23.718271 2041 log.cc:1196] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/d84777f46a514c02bbbb4f216739f42a
I20260927 12:35:23.718320 2042 ts_tablet_manager.cc:1918] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:23.718396 2042 log.cc:1196] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/d84777f46a514c02bbbb4f216739f42a
I20260927 12:35:23.718498 2041 ts_tablet_manager.cc:1939] T d84777f46a514c02bbbb4f216739f42a P 48a2d324c14c4eacaea977c92e596ed1: Deleting consensus metadata
I20260927 12:35:23.718645 2042 ts_tablet_manager.cc:1939] T d84777f46a514c02bbbb4f216739f42a P 119297e7373c403296608d3913538cd4: Deleting consensus metadata
I20260927 12:35:23.719377 1498 catalog_manager.cc:4841] TS 119297e7373c403296608d3913538cd4 (127.0.98.3:33347): tablet d84777f46a514c02bbbb4f216739f42a (table test-table [id=23d9d8bca8e84dc0bd4f2d16ed638fc2]) successfully deleted
I20260927 12:35:23.719756 1498 catalog_manager.cc:4841] TS 48a2d324c14c4eacaea977c92e596ed1 (127.0.98.1:38003): tablet d84777f46a514c02bbbb4f216739f42a (table test-table [id=23d9d8bca8e84dc0bd4f2d16ed638fc2]) successfully deleted
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512523, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:23.733246 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.719942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.98.1:38003)
0927 12:35:23.720057 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.720062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.720456 (+ 394us) server_negotiation.cc:407] Connection header received
0927 12:35:23.720687 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.720690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.720737 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.720780 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:23.721796 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.722592 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.723979 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.724356 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.724482 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.728637 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:23.728655 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:23.728676 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:23.728718 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:23.731001 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.731682 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.731690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.731692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.731815 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.732155 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.732159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.732161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.732489 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:23.732708 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.732926 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.733070 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 12:35:23.733691 1692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35210
I20260927 12:35:23.745290 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:23.735632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.0.98.62:34319)
0927 12:35:23.735774 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:23.735779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:23.736274 (+ 495us) server_negotiation.cc:407] Connection header received
0927 12:35:23.736385 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:23.736388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:23.736444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:23.736553 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:23.737362 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.738135 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.739284 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:23.739634 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:23.739753 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:23.741507 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:23.741511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:23.741514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:23.741551 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:23.743307 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.743829 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.743833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.743834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.743940 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:23.744296 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:23.744300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:23.744302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:23.744573 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:23.744715 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:23.744980 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 12:35:23.745111 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 12:35:23.746026 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55000
I20260927 12:35:23.747605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1583
I20260927 12:35:23.756803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1723
I20260927 12:35:23.762910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1863
I20260927 12:35:23.768057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 1479
2026-09-27T12:35:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3933 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[2080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[2080](info): set up 2 sockets
Sep 27 12:35:23 dist-test-slave-tzgx krb5kdc[2080](info): commencing operation
krb5kdc: starting...
W20260927 12:35:24.735678 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:24 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:24.857535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:24.971029 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:24.971340 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:24.971402 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:24.975750 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:24.975835 2093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:35:24.975865 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:24.975888 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:24.975912 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:24.975939 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:24.981629 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45363
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:24.982781 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:24.983886 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:24.992664 2101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:24.992650 2099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:24.992653 2098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:24.992913 2093 server_base.cc:1034] running on GCE node
I20260927 12:35:24.993623 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:24.994786 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:24.996001 2093 hybrid_clock.cc:648] HybridClock initialized: now 1790512524995979 us; error 55 us; skew 500 ppm
Sep 27 12:35:24 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:24.998922 2093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:24.999945 2093 webserver.cc:460] Webserver started at http://127.0.98.62:41563/ using document root <none> and password file <none>
I20260927 12:35:25.000557 2093 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:25.000625 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:25.000826 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:25.002935 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "dea29c1ea771429fa04d31ef06f193f2"
format_stamp: "Formatted at 2026-09-27 12:35:24 on dist-test-slave-tzgx"
server_key: "4b2e8095de0670cf052162663a13e58a"
server_key_iv: "e5de59ec6839fd6209c5808919688cae"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.003554 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "dea29c1ea771429fa04d31ef06f193f2"
format_stamp: "Formatted at 2026-09-27 12:35:24 on dist-test-slave-tzgx"
server_key: "4b2e8095de0670cf052162663a13e58a"
server_key_iv: "e5de59ec6839fd6209c5808919688cae"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.007449 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:25.010267 2108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.011706 2093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:25.011850 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "dea29c1ea771429fa04d31ef06f193f2"
format_stamp: "Formatted at 2026-09-27 12:35:24 on dist-test-slave-tzgx"
server_key: "4b2e8095de0670cf052162663a13e58a"
server_key_iv: "e5de59ec6839fd6209c5808919688cae"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.012012 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:25.023880 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:25.027251 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:25.027452 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:25.036465 2093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45363
I20260927 12:35:25.036494 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45363 every 8 connection(s)
I20260927 12:35:25.037456 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:25.040416 2171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:25.044044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 2093
I20260927 12:35:25.044200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:25.044432 392 external_mini_cluster.cc:1428] Setting key 6104aabff42c5ae52f0b484c1039cfa0
I20260927 12:35:25.046520 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Bootstrap starting.
I20260927 12:35:25.048815 2171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:25.049551 2171 log.cc:824] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:25.051664 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: No bootstrap required, opened a new log
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:25.055760 2171 raft_consensus.cc:348] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea29c1ea771429fa04d31ef06f193f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45363 } }
I20260927 12:35:25.056118 2171 raft_consensus.cc:374] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:25.056212 2171 raft_consensus.cc:729] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea29c1ea771429fa04d31ef06f193f2, State: Initialized, Role: FOLLOWER
I20260927 12:35:25.056778 2171 consensus_queue.cc:260] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea29c1ea771429fa04d31ef06f193f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45363 } }
I20260927 12:35:25.056937 2171 raft_consensus.cc:388] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:25.057006 2171 raft_consensus.cc:482] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:25.057107 2171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:25.058220 2171 raft_consensus.cc:504] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea29c1ea771429fa04d31ef06f193f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45363 } }
I20260927 12:35:25.058579 2171 leader_election.cc:302] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dea29c1ea771429fa04d31ef06f193f2; no voters:
I20260927 12:35:25.058969 2171 leader_election.cc:288] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:25.058970 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.045837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.98.62:45363)
0927 12:35:25.046318 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.046334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.046372 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:35:25.047064 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.047084 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.047441 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.047761 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.048563 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.049584 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.050625 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.051110 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.051222 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.053984 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.053989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.054003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.054032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.055646 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.056230 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.056237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.056245 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.056401 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.056753 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.056756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.056758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.057263 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.057414 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.057712 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.058034 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":160,"threads_started":1}
I20260927 12:35:25.059073 2176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:25.059267 2176 raft_consensus.cc:686] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [term 1 LEADER]: Becoming Leader. State: Replica: dea29c1ea771429fa04d31ef06f193f2, State: Running, Role: LEADER
I20260927 12:35:25.059646 2176 consensus_queue.cc:237] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea29c1ea771429fa04d31ef06f193f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45363 } }
I20260927 12:35:25.060305 2171 sys_catalog.cc:564] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:25.062212 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea29c1ea771429fa04d31ef06f193f2. Latest consensus state: current_term: 1 leader_uuid: "dea29c1ea771429fa04d31ef06f193f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea29c1ea771429fa04d31ef06f193f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45363 } } }
I20260927 12:35:25.062350 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:25.062788 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dea29c1ea771429fa04d31ef06f193f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dea29c1ea771429fa04d31ef06f193f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45363 } } }
I20260927 12:35:25.063145 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2 [sys.catalog]: This master's current role is: LEADER
W20260927 12:35:25.065326 2192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:25.065404 2192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:25.065490 2191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:25.066411 2191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:25.073374 2191 catalog_manager.cc:1347] Generated new cluster ID: b00652d667ab49a89e39ebe3535d67b1
I20260927 12:35:25.073462 2191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:25.083382 2191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:25.084460 2191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:25.093717 2191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Generated new TSK 0
I20260927 12:35:25.094565 2191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:25.127794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45363
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:25.240053 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:25.240386 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:25.240434 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:25.244513 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:25.244575 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:25.244616 2197 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:35:25.244629 2197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:25.244660 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:25.250072 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:25.251143 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:25.252142 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:25.262614 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:25.262603 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:25.262672 2197 server_base.cc:1034] running on GCE node
W20260927 12:35:25.262603 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:25.263378 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:25.264293 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:25.265483 2197 hybrid_clock.cc:648] HybridClock initialized: now 1790512525265462 us; error 37 us; skew 500 ppm
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:25.268280 2197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:25.269186 2197 webserver.cc:460] Webserver started at http://127.0.98.1:44999/ using document root <none> and password file <none>
I20260927 12:35:25.269836 2197 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:25.269958 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:25.270211 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:25.272404 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "26586d97a37240f083da6bead804f78f"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "4c3570eeeeb9c6092b23ff62c93931c1"
server_key_iv: "479d7a4aed31ce270d9f3c9a70b6ef72"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.272948 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26586d97a37240f083da6bead804f78f"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "4c3570eeeeb9c6092b23ff62c93931c1"
server_key_iv: "479d7a4aed31ce270d9f3c9a70b6ef72"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.276844 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:25.279417 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.280792 2197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:35:25.280913 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "26586d97a37240f083da6bead804f78f"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "4c3570eeeeb9c6092b23ff62c93931c1"
server_key_iv: "479d7a4aed31ce270d9f3c9a70b6ef72"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.281018 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:25.299674 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:25.303022 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:25.303181 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:25.304680 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:25.304738 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.304792 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:25.304819 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:25.314472 2197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33019
I20260927 12:35:25.314610 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33019 every 8 connection(s)
I20260927 12:35:25.315593 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:25.317847 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.305515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36229 (local address 127.0.98.62:45363)
0927 12:35:25.305658 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.305664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.306580 (+ 916us) server_negotiation.cc:407] Connection header received
0927 12:35:25.307611 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.307615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.307669 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.307743 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.309268 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.310088 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.311356 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.311599 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.311689 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.314225 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.314229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.314232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.314262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.315658 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.316268 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.316273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.316275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.316368 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.316739 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.316743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.316745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.316945 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.317032 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.317557 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.317686 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:35:25.318583 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.305930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.98.1:36229)
0927 12:35:25.306402 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.306454 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.307382 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.307943 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.307953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.308397 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:25.309088 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.309111 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.310229 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.310233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.311233 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.311243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.311743 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.311747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.311921 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.312577 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:25.312598 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:25.314058 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:25.315766 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.315771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.315788 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.316174 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.316454 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.316458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.316460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.316631 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.317048 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:25.317055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:25.317292 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.317446 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.317772 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260927 12:35:25.319736 2328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45363
I20260927 12:35:25.319864 2328 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:25.320395 2328 heartbeater.cc:511] Master 127.0.98.62:45363 requested a full tablet report, sending...
I20260927 12:35:25.321935 2123 ts_manager.cc:194] Registered new tserver with Master: 26586d97a37240f083da6bead804f78f (127.0.98.1:33019)
I20260927 12:35:25.323215 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36229
I20260927 12:35:25.324878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 2197
I20260927 12:35:25.324985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:25.325282 392 external_mini_cluster.cc:1428] Setting key 661f5ac4c493ec230109d548e3131beb
I20260927 12:35:25.328471 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.321857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35707 (local address 127.0.98.62:45363)
0927 12:35:25.321968 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.321972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.322160 (+ 188us) server_negotiation.cc:407] Connection header received
0927 12:35:25.322196 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.322199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.322236 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.322298 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.322748 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.323246 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.324174 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.324394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.324476 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.325663 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.325667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.325670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.325694 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.327082 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.327438 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.327441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.327442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.327527 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.327729 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.327731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.327733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.327906 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.327996 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.328268 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.328354 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:25.328552 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.321768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.98.1:35707)
0927 12:35:25.322030 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.322044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.322138 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.322325 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.322328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.322533 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:25.322649 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.322656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.323359 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.323363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.324036 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.324045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.324492 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.324496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.324597 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.325096 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:25.325109 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:25.325517 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:25.327187 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.327190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.327192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.327365 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.327597 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.327599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.327601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.327663 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.327990 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:25.328003 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:25.328120 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.328223 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.328355 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:25.350627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45363
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:25.461882 2337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:25.462184 2337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:25.462255 2337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:25.466529 2337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:25.466605 2337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:25.466667 2337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:35:25.466694 2337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:25.466748 2337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:25.473177 2337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:25.474328 2337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:25.475395 2337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:25.485263 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:25.485230 2342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:25.485258 2343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:25.485417 2337 server_base.cc:1034] running on GCE node
I20260927 12:35:25.485965 2337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:25.486646 2337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:25.487808 2337 hybrid_clock.cc:648] HybridClock initialized: now 1790512525487774 us; error 52 us; skew 500 ppm
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:25.490787 2337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:25.491751 2337 webserver.cc:460] Webserver started at http://127.0.98.2:39563/ using document root <none> and password file <none>
I20260927 12:35:25.492357 2337 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:25.492426 2337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:25.492645 2337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:25.494688 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa705e98965743558b77d9b74dc3e9e2"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "059be0e15438f5907036ae3ab88b8af4"
server_key_iv: "2a1c4fbdf927820e1083911814c68020"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.495257 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa705e98965743558b77d9b74dc3e9e2"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "059be0e15438f5907036ae3ab88b8af4"
server_key_iv: "2a1c4fbdf927820e1083911814c68020"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.499159 2337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:35:25.501930 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.503239 2337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:25.503364 2337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "fa705e98965743558b77d9b74dc3e9e2"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "059be0e15438f5907036ae3ab88b8af4"
server_key_iv: "2a1c4fbdf927820e1083911814c68020"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.503515 2337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:25.526108 2337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:25.529145 2337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:25.529325 2337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:25.530757 2337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:25.530807 2337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.530859 2337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:25.530889 2337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:25.540647 2337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43501
I20260927 12:35:25.540783 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43501 every 8 connection(s)
I20260927 12:35:25.541652 2337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:25.543987 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.531597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53537 (local address 127.0.98.62:45363)
0927 12:35:25.531711 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.531715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.532837 (+ 1122us) server_negotiation.cc:407] Connection header received
0927 12:35:25.533766 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.533771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.533826 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.533881 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.535305 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.535979 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.537264 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.537586 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.537695 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.540844 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.540848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.540851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.540881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.542220 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.542764 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.542767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.542768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.542846 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.543115 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.543117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.543118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.543277 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.543346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.543768 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.543867 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:35:25.544773 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.532193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.98.2:53537)
0927 12:35:25.532684 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.532728 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.533529 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.534286 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.534293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.534639 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:25.535163 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.535182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.536128 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.536131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.537152 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.537161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.537743 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.537747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.537909 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.538975 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:25.538993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:25.540689 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:25.542337 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.542342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.542356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.542668 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.542916 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.542918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.542919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.543049 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.543344 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:25.543348 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:25.543582 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.543723 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.543964 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260927 12:35:25.545861 2468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45363
I20260927 12:35:25.546008 2468 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:25.546547 2468 heartbeater.cc:511] Master 127.0.98.62:45363 requested a full tablet report, sending...
I20260927 12:35:25.547164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 2337
I20260927 12:35:25.547273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:25.547577 392 external_mini_cluster.cc:1428] Setting key 2fb1cacb7e12dfba5a1c841092a1a0de
I20260927 12:35:25.547683 2125 ts_manager.cc:194] Registered new tserver with Master: fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501)
I20260927 12:35:25.548437 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53537
I20260927 12:35:25.555685 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.548046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.98.2:41661)
0927 12:35:25.548484 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.548499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.548623 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.549237 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.549241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.549443 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:25.549568 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.549575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.550506 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.550509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.551073 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.551081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.551585 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.551587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.551653 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.552195 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:25.552207 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:25.552531 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:25.554284 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.554287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.554289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.554480 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.554742 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.554744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.554746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.554816 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.555257 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:25.555259 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:25.555333 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.555406 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.555518 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":98,"threads_started":1}
I20260927 12:35:25.555773 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.548203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41661 (local address 127.0.98.62:45363)
0927 12:35:25.548321 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.548325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.548561 (+ 236us) server_negotiation.cc:407] Connection header received
0927 12:35:25.549072 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.549075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.549124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.549204 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.549703 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.550390 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.551182 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.551488 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.551586 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.552668 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.552672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.552675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.552702 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.554164 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.554572 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.554576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.554578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.554660 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.554892 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.554895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.554897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.555175 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.555277 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.555480 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.555612 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:25.574608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45363
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:25.684213 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:25.684532 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:25.684587 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:25.688638 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:25.688699 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:25.688740 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:35:25.688752 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:25.688784 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:25.694231 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:25.695273 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:25.696311 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:25.705706 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:25.705701 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:25.705747 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:25.706012 2477 server_base.cc:1034] running on GCE node
I20260927 12:35:25.706427 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:25.707098 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:25.708246 2477 hybrid_clock.cc:648] HybridClock initialized: now 1790512525708225 us; error 30 us; skew 500 ppm
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:25.710920 2477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:25.711768 2477 webserver.cc:460] Webserver started at http://127.0.98.3:33909/ using document root <none> and password file <none>
I20260927 12:35:25.712352 2477 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:25.712419 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:25.712632 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:25.714669 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "8640c3820bee400e90b93792fae16472"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "380cdb10549da442f0875af16de43830"
server_key_iv: "418b24f00ecb445cdf04a71b53eee9b4"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.715233 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8640c3820bee400e90b93792fae16472"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "380cdb10549da442f0875af16de43830"
server_key_iv: "418b24f00ecb445cdf04a71b53eee9b4"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.718993 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:35:25.721787 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.723032 2477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:25.723150 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "8640c3820bee400e90b93792fae16472"
format_stamp: "Formatted at 2026-09-27 12:35:25 on dist-test-slave-tzgx"
server_key: "380cdb10549da442f0875af16de43830"
server_key_iv: "418b24f00ecb445cdf04a71b53eee9b4"
server_key_version: "encryptionkey@0"
I20260927 12:35:25.723256 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:25.746248 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:25.749261 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:25.749444 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:25.750906 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:25.750957 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.751008 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:25.751036 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:25.760754 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39707
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:25.760901 2607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39707 every 8 connection(s)
I20260927 12:35:25.761642 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:25.764945 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.751758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48501 (local address 127.0.98.62:45363)
0927 12:35:25.751897 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.751901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.753005 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 12:35:25.754070 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.754074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.754130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.754216 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.755789 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.756626 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.758288 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.758508 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.758591 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.761468 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.761473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.761475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.761511 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.762908 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.763436 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.763439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.763440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.763541 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.763828 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.763832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.763833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.764036 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.764161 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.764684 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.764790 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:35:25.765715 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.752362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.98.3:48501)
0927 12:35:25.752850 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.752893 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.753835 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.754437 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.754448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.754894 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:25.755618 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.755642 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.756752 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.756756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.758173 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.758183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.758653 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.758658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.758867 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.759574 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:25.759593 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:25.761301 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:25.763042 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.763047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.763059 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.763316 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.763612 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.763614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.763616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.763732 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.764196 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:25.764202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:25.764446 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.764613 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.764873 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260927 12:35:25.766947 2608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45363
I20260927 12:35:25.767081 2608 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:25.767642 2608 heartbeater.cc:511] Master 127.0.98.62:45363 requested a full tablet report, sending...
I20260927 12:35:25.768813 2125 ts_manager.cc:194] Registered new tserver with Master: 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:25.769502 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48501
I20260927 12:35:25.770905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 2477
I20260927 12:35:25.771009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:25.771266 392 external_mini_cluster.cc:1428] Setting key 1226f13a7eb78e68daad70db47ce121a
I20260927 12:35:25.773149 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:35:25.776145 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.769201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33773 (local address 127.0.98.62:45363)
0927 12:35:25.769315 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.769319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.769439 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:35:25.769531 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.769535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.769581 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.769669 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.770143 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.770626 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.771656 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.771959 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.772050 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.773141 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.773144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.773147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.773172 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.774578 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.775004 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.775008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.775010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.775106 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.775346 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.775349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.775351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.775531 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.775626 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.775865 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.775970 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:35:25.776155 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.769117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.98.3:33773)
0927 12:35:25.769361 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.769373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.769460 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.769688 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.769691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.769917 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:25.770031 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.770038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.770749 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.770753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.771532 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.771544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.772063 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.772066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.772183 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.772664 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:25.772677 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:25.773023 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:25.774683 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.774686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.774687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.774873 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.775188 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:25.775191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:25.775193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:25.775270 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:25.775626 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:25.775629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:25.775719 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.775804 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.775961 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:35:25 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:25.798171 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.790349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.98.62:45363)
0927 12:35:25.790472 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.790476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.790487 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:35:25.790530 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.790533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.790574 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.790649 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:25.791342 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.791894 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.792831 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.793059 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.793151 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.795473 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:25.795477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:25.795479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:25.795506 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:25.796851 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.797219 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.797222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.797223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.797293 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:25.797487 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:25.797489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:25.797491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:25.797653 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:25.797743 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.797905 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.798012 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:35:25.800985 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:25.803982 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:25.818780 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.812901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.0.98.1:33019)
0927 12:35:25.813827 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.813843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.813866 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:25.813943 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.813949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.814155 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.814289 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:25.814617 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.815836 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.817478 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.818127 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.818226 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.818395 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.818525 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.818624 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":75,"threads_started":1}
I20260927 12:35:25.818769 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.812770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33019 (local address 127.0.0.1:44770)
0927 12:35:25.813340 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.813389 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.813535 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.814382 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.814388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.814410 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:25.814513 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.814518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.815945 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.815949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.817352 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.817361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.818238 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.818242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.818341 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.818375 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.818488 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.818619 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":102,"thread_start_us":89,"threads_started":1}
I20260927 12:35:25.819006 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.812872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43501 (local address 127.0.0.1:32816)
0927 12:35:25.813618 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.813630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.813699 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.814862 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.814865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.814881 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:25.814954 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.814959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.816160 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.816163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.817531 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.817541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.818629 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.818632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.818716 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.818736 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.818786 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.818855 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":71,"thread_start_us":94,"threads_started":1}
I20260927 12:35:25.819139 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.813085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39707 (local address 127.0.0.1:49970)
0927 12:35:25.813906 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:25.813917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:25.814005 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:25.814578 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:25.814581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:25.814597 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:25.814692 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.814698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.816033 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.816037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.817531 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:25.817541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.818821 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.818824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:25.818898 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.818917 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:35:25.818975 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 12:35:25.819033 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":114,"thread_start_us":86,"threads_started":1}
I20260927 12:35:25.819543 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.813589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.0.98.3:39707)
0927 12:35:25.814003 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.814014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.814037 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:25.814107 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.814112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.814299 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.814442 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:25.814808 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.815929 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.817667 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.818491 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.818589 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.818768 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.819295 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.819390 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":93,"threads_started":1}
I20260927 12:35:25.819669 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:25.813590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.0.98.2:43501)
0927 12:35:25.814446 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:25.814457 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:25.814476 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:35:25.814555 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:25.814560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:25.814729 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:25.815108 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:25.815217 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.816041 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.817667 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:25.818485 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:25.819257 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:25.819416 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:25.819459 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 12:35:25.819541 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":85,"threads_started":1}
I20260927 12:35:25.822067 2262 tablet_service.cc:1467] Processing CreateTablet for tablet e259ec06373c4d26a305b0fc04298ad5 (DEFAULT_TABLE table=test-table [id=f8abc440f10349de823b60f3dc42c00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:25.822284 2542 tablet_service.cc:1467] Processing CreateTablet for tablet e259ec06373c4d26a305b0fc04298ad5 (DEFAULT_TABLE table=test-table [id=f8abc440f10349de823b60f3dc42c00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:25.822288 2402 tablet_service.cc:1467] Processing CreateTablet for tablet e259ec06373c4d26a305b0fc04298ad5 (DEFAULT_TABLE table=test-table [id=f8abc440f10349de823b60f3dc42c00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:25.823093 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259ec06373c4d26a305b0fc04298ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:25.823302 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259ec06373c4d26a305b0fc04298ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:25.823297 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259ec06373c4d26a305b0fc04298ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:25.829916 2631 tablet_bootstrap.cc:492] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Bootstrap starting.
I20260927 12:35:25.831194 2632 tablet_bootstrap.cc:492] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Bootstrap starting.
I20260927 12:35:25.832451 2633 tablet_bootstrap.cc:492] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Bootstrap starting.
I20260927 12:35:25.833273 2631 tablet_bootstrap.cc:654] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:25.833273 2632 tablet_bootstrap.cc:654] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:25.834220 2632 log.cc:824] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:25.834487 2633 tablet_bootstrap.cc:654] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:25.834859 2631 log.cc:824] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:25.835266 2633 log.cc:824] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:25.836470 2632 tablet_bootstrap.cc:492] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: No bootstrap required, opened a new log
I20260927 12:35:25.836697 2632 ts_tablet_manager.cc:1397] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 12:35:25.836854 2631 tablet_bootstrap.cc:492] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: No bootstrap required, opened a new log
I20260927 12:35:25.837052 2631 ts_tablet_manager.cc:1397] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:35:25.837157 2633 tablet_bootstrap.cc:492] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: No bootstrap required, opened a new log
I20260927 12:35:25.837340 2633 ts_tablet_manager.cc:1397] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:35:25.840466 2633 raft_consensus.cc:348] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:25.840713 2633 raft_consensus.cc:374] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:25.840765 2633 raft_consensus.cc:729] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Initialized, Role: FOLLOWER
I20260927 12:35:25.840626 2632 raft_consensus.cc:348] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:25.840888 2632 raft_consensus.cc:374] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:25.840950 2632 raft_consensus.cc:729] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8640c3820bee400e90b93792fae16472, State: Initialized, Role: FOLLOWER
I20260927 12:35:25.840898 2631 raft_consensus.cc:348] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:25.841161 2631 raft_consensus.cc:374] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:25.841215 2631 raft_consensus.cc:729] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26586d97a37240f083da6bead804f78f, State: Initialized, Role: FOLLOWER
I20260927 12:35:25.841262 2633 consensus_queue.cc:260] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:25.841377 2632 consensus_queue.cc:260] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:25.841563 2631 consensus_queue.cc:260] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:25.842044 2468 heartbeater.cc:503] Master 127.0.98.62:45363 was elected leader, sending a full tablet report...
I20260927 12:35:25.842300 2633 ts_tablet_manager.cc:1428] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:35:25.842337 2632 ts_tablet_manager.cc:1428] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:35:25.842679 2608 heartbeater.cc:503] Master 127.0.98.62:45363 was elected leader, sending a full tablet report...
I20260927 12:35:25.842939 2328 heartbeater.cc:503] Master 127.0.98.62:45363 was elected leader, sending a full tablet report...
I20260927 12:35:25.843528 2631 ts_tablet_manager.cc:1428] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 12:35:26.012606 2609 tablet.cc:2369] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:26.042835 2469 tablet.cc:2369] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:26.067054 2329 tablet.cc:2369] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:26.210253 2637 raft_consensus.cc:482] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:26.210479 2637 raft_consensus.cc:504] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.211648 2637 leader_election.cc:288] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:26.217108 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.212062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47569 (local address 127.0.98.3:39707)
0927 12:35:26.212199 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.212204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.212385 (+ 181us) server_negotiation.cc:407] Connection header received
0927 12:35:26.212545 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.212547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.212597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.212677 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:26.213111 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.213891 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.215748 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.216451 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.216563 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.216630 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.216927 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.217000 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:35:26.217133 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.211993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59315 (local address 127.0.98.1:33019)
0927 12:35:26.212146 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.212150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.212387 (+ 237us) server_negotiation.cc:407] Connection header received
0927 12:35:26.212539 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.212542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.212597 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.212677 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:26.213111 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.213891 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.215792 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.216451 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.216563 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.216630 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.216938 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.217016 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:35:26.217159 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.211943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39707 (local address 127.0.98.2:47569)
0927 12:35:26.212288 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.212302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.212434 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.212733 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.212736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.212767 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:26.213018 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.213024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.214010 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.214013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.215582 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.215596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.216576 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.216580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.216754 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.216811 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.216899 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.216985 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":286,"spinlock_wait_cycles":28416,"thread_start_us":107,"threads_started":1}
I20260927 12:35:26.217521 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.211870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33019 (local address 127.0.98.2:59315)
0927 12:35:26.212289 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.212302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.212434 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.212733 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.212736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.212767 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:26.212998 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.213006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.214001 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.214004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.215582 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.215596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.216576 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.216580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.216759 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.216810 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.217325 (+ 515us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.217405 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":232,"spinlock_wait_cycles":17920,"thread_start_us":94,"threads_started":1}
I20260927 12:35:26.217998 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259ec06373c4d26a305b0fc04298ad5" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472" is_pre_election: true
I20260927 12:35:26.218070 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259ec06373c4d26a305b0fc04298ad5" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f" is_pre_election: true
I20260927 12:35:26.218287 2562 raft_consensus.cc:2454] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 0.
I20260927 12:35:26.218320 2282 raft_consensus.cc:2454] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 0.
I20260927 12:35:26.218742 2354 leader_election.cc:302] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:26.219013 2637 raft_consensus.cc:2781] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:26.219097 2637 raft_consensus.cc:482] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:26.219144 2637 raft_consensus.cc:3037] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.220206 2637 raft_consensus.cc:504] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.220546 2637 leader_election.cc:288] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 election: Requested vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:26.220913 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259ec06373c4d26a305b0fc04298ad5" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f"
I20260927 12:35:26.220913 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259ec06373c4d26a305b0fc04298ad5" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472"
I20260927 12:35:26.221051 2562 raft_consensus.cc:3037] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.221063 2282 raft_consensus.cc:3037] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.222306 2282 raft_consensus.cc:2454] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 1.
I20260927 12:35:26.222306 2562 raft_consensus.cc:2454] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 1.
I20260927 12:35:26.222620 2355 leader_election.cc:302] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8640c3820bee400e90b93792fae16472, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:26.222795 2637 raft_consensus.cc:2781] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:26.222975 2637 raft_consensus.cc:686] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 1 LEADER]: Becoming Leader. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Running, Role: LEADER
I20260927 12:35:26.223311 2637 consensus_queue.cc:237] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.226430 2123 catalog_manager.cc:5495] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa705e98965743558b77d9b74dc3e9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } health_report { overall_health: UNKNOWN } } }
Sep 27 12:35:26 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:26.305689 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.297824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45363 (local address 127.0.0.1:55720)
0927 12:35:26.298177 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.298189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.298302 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.298525 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.298528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.298933 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:26.299025 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.299032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.299721 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.299724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.300321 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.300330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.300716 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.300720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.300834 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.301354 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:26.301365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:26.302719 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:26.304220 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:26.304230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:26.304237 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:26.304454 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:26.304685 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:26.304688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:26.304690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:26.304761 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:26.305123 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:26.305131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:26.305237 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.305329 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.305492 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":30,"thread_start_us":130,"threads_started":1}
I20260927 12:35:26.305689 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.297938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.0.98.62:45363)
0927 12:35:26.298082 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.298087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.298261 (+ 174us) server_negotiation.cc:407] Connection header received
0927 12:35:26.298379 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.298382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.298418 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.298480 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:26.299132 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.299628 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.300419 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.300622 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.300706 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.302840 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:26.302843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:26.302845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:26.302865 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:26.304137 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:26.304542 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:26.304545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:26.304547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:26.304614 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:26.304838 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:26.304842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:26.304844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:26.305029 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:26.305111 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.305383 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.305543 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":48}
I20260927 12:35:26.309919 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:35:26.310690 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:26.315732 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 2547653468894c089131e4acc1a2bbcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e881c3401014a2b8c7f87e1a0623153]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:26.315732 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 2547653468894c089131e4acc1a2bbcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e881c3401014a2b8c7f87e1a0623153]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:26.315764 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 2547653468894c089131e4acc1a2bbcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6e881c3401014a2b8c7f87e1a0623153]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:26.316525 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2547653468894c089131e4acc1a2bbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:26.316525 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2547653468894c089131e4acc1a2bbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:26.316525 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2547653468894c089131e4acc1a2bbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:26.319469 2631 tablet_bootstrap.cc:492] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: Bootstrap starting.
I20260927 12:35:26.319878 2632 tablet_bootstrap.cc:492] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: Bootstrap starting.
I20260927 12:35:26.320528 2631 tablet_bootstrap.cc:654] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:26.320814 2632 tablet_bootstrap.cc:654] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:26.321724 2633 tablet_bootstrap.cc:492] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: Bootstrap starting.
I20260927 12:35:26.322118 2632 tablet_bootstrap.cc:492] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: No bootstrap required, opened a new log
I20260927 12:35:26.322216 2632 ts_tablet_manager.cc:1397] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:35:26.322695 2632 raft_consensus.cc:348] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.322821 2632 raft_consensus.cc:374] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:26.322861 2632 raft_consensus.cc:729] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8640c3820bee400e90b93792fae16472, State: Initialized, Role: FOLLOWER
I20260927 12:35:26.322947 2633 tablet_bootstrap.cc:654] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:26.322997 2632 consensus_queue.cc:260] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.323299 2632 ts_tablet_manager.cc:1428] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:26.323590 2638 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.323727 2638 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.324595 2631 tablet_bootstrap.cc:492] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: No bootstrap required, opened a new log
I20260927 12:35:26.324671 2631 ts_tablet_manager.cc:1397] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 12:35:26.325016 2633 tablet_bootstrap.cc:492] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: No bootstrap required, opened a new log
I20260927 12:35:26.325110 2633 ts_tablet_manager.cc:1397] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:26.325049 2631 raft_consensus.cc:348] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.325160 2631 raft_consensus.cc:374] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:26.325196 2631 raft_consensus.cc:729] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26586d97a37240f083da6bead804f78f, State: Initialized, Role: FOLLOWER
I20260927 12:35:26.325325 2631 consensus_queue.cc:260] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.325565 2633 raft_consensus.cc:348] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.325580 2639 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.325683 2633 raft_consensus.cc:374] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:26.325713 2639 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.325717 2633 raft_consensus.cc:729] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Initialized, Role: FOLLOWER
I20260927 12:35:26.325832 2633 consensus_queue.cc:260] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.326071 2633 ts_tablet_manager.cc:1428] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:26.325580 2631 ts_tablet_manager.cc:1428] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:35:26.326292 2637 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.326427 2637 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.361500 2638 raft_consensus.cc:482] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:26.361644 2638 raft_consensus.cc:504] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.362819 2638 leader_election.cc:288] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501)
I20260927 12:35:26.366904 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.363165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52061 (local address 127.0.98.1:33019)
0927 12:35:26.363297 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.363302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.363482 (+ 180us) server_negotiation.cc:407] Connection header received
0927 12:35:26.363644 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.363646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.363688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.363748 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:26.364010 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.364548 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.365904 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.366298 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.366390 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.366456 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.366703 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.366776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:35:26.366925 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.363242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45933 (local address 127.0.98.2:43501)
0927 12:35:26.363442 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.363458 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.363474 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:26.363645 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.363648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.363689 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.363759 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:26.364021 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.364571 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.365896 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.366344 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.366444 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.366534 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.366696 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.366771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":56,"threads_started":1}
I20260927 12:35:26.366930 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.363065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43501 (local address 127.0.98.3:45933)
0927 12:35:26.363385 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.363399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.363529 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.363782 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.363784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.363799 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:26.363919 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.363923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.364676 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.364679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.365786 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.365795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.366445 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.366448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.366566 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.366608 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.366674 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.366768 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":33,"spinlock_wait_cycles":27904,"thread_start_us":79,"threads_started":1}
I20260927 12:35:26.366933 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.363054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33019 (local address 127.0.98.3:52061)
0927 12:35:26.363376 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.363391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.363533 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.363768 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.363771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.363785 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:26.363905 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.363911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.364658 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.364661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.365795 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.365804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.366440 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.366443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.366570 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.366615 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.366677 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.366776 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":88,"spinlock_wait_cycles":2304,"thread_start_us":82,"threads_started":1}
I20260927 12:35:26.367576 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2547653468894c089131e4acc1a2bbcf" candidate_uuid: "8640c3820bee400e90b93792fae16472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa705e98965743558b77d9b74dc3e9e2" is_pre_election: true
I20260927 12:35:26.367743 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2547653468894c089131e4acc1a2bbcf" candidate_uuid: "8640c3820bee400e90b93792fae16472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f" is_pre_election: true
I20260927 12:35:26.367834 2422 raft_consensus.cc:2454] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8640c3820bee400e90b93792fae16472 in term 0.
I20260927 12:35:26.367867 2282 raft_consensus.cc:2454] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8640c3820bee400e90b93792fae16472 in term 0.
I20260927 12:35:26.368242 2494 leader_election.cc:302] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, 8640c3820bee400e90b93792fae16472, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:26.368507 2638 raft_consensus.cc:2781] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:26.368564 2638 raft_consensus.cc:482] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:26.368590 2638 raft_consensus.cc:3037] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.369549 2638 raft_consensus.cc:504] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.369882 2638 leader_election.cc:288] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [CANDIDATE]: Term 1 election: Requested vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501)
I20260927 12:35:26.370210 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2547653468894c089131e4acc1a2bbcf" candidate_uuid: "8640c3820bee400e90b93792fae16472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f"
I20260927 12:35:26.370316 2282 raft_consensus.cc:3037] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.370338 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2547653468894c089131e4acc1a2bbcf" candidate_uuid: "8640c3820bee400e90b93792fae16472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa705e98965743558b77d9b74dc3e9e2"
I20260927 12:35:26.370456 2422 raft_consensus.cc:3037] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.371178 2282 raft_consensus.cc:2454] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8640c3820bee400e90b93792fae16472 in term 1.
I20260927 12:35:26.371371 2422 raft_consensus.cc:2454] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8640c3820bee400e90b93792fae16472 in term 1.
I20260927 12:35:26.371444 2494 leader_election.cc:302] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, 8640c3820bee400e90b93792fae16472; no voters:
I20260927 12:35:26.371618 2638 raft_consensus.cc:2781] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:26.371807 2638 raft_consensus.cc:686] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [term 1 LEADER]: Becoming Leader. State: Replica: 8640c3820bee400e90b93792fae16472, State: Running, Role: LEADER
I20260927 12:35:26.372215 2638 consensus_queue.cc:237] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.374360 2656 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: TxnStatusTablet state changed. Reason: New leader 8640c3820bee400e90b93792fae16472. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.374495 2656 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:26.374751 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:35:26.374953 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:35:26.375715 2118 catalog_manager.cc:5495] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8640c3820bee400e90b93792fae16472 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } health_report { overall_health: HEALTHY } } }
I20260927 12:35:26.399483 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.395127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.0.98.3:39707)
0927 12:35:26.395271 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.395276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.395370 (+ 94us) server_negotiation.cc:407] Connection header received
0927 12:35:26.396158 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.396161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.396227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.396336 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:26.396608 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.397118 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.397990 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.398208 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.398306 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.398626 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.399067 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.399181 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.399268 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.399326 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:26.399483 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.395005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39707 (local address 127.0.0.1:49974)
0927 12:35:26.395284 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.395299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.395436 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.396369 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.396373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.396381 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:26.396496 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.396504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.397233 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.397237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.397866 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.397875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.398325 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.398330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.398426 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.398474 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.399169 (+ 695us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.399183 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.399257 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.399356 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 12:35:26.405051 2282 raft_consensus.cc:1261] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Refusing update from remote peer 8640c3820bee400e90b93792fae16472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:26.405042 2422 raft_consensus.cc:1261] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Refusing update from remote peer 8640c3820bee400e90b93792fae16472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:26.405805 2638 consensus_queue.cc:1035] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:26.406013 2656 consensus_queue.cc:1035] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:26.409421 2637 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: TxnStatusTablet state changed. Reason: New leader 8640c3820bee400e90b93792fae16472. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.409564 2637 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.409981 2639 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: TxnStatusTablet state changed. Reason: New leader 8640c3820bee400e90b93792fae16472. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.410097 2639 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.412824 2656 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.412971 2656 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:26.413084 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333939308147539968 to 7333939308023271424. Current Snapshot: MvccSnapshot[applied={T|T < 7333939308147539968}]
I20260927 12:35:26.414551 2637 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.414661 2637 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P fa705e98965743558b77d9b74dc3e9e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.415406 2663 mvcc.cc:205] Tried to move back new op lower bound from 7333939308147539968 to 7333939308023271424. Current Snapshot: MvccSnapshot[applied={T|T < 7333939308147539968}]
I20260927 12:35:26.415817 2656 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.415941 2656 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 8640c3820bee400e90b93792fae16472: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:26.418632 2639 tablet_replica.cc:439] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8640c3820bee400e90b93792fae16472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } } }
I20260927 12:35:26.418769 2639 tablet_replica.cc:442] T 2547653468894c089131e4acc1a2bbcf P 26586d97a37240f083da6bead804f78f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:26.429512 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.424848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.0.98.2:43501)
0927 12:35:26.424990 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.424996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.425121 (+ 125us) server_negotiation.cc:407] Connection header received
0927 12:35:26.425275 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.425278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.425372 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.425506 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:26.425787 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.426544 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.427464 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.427839 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.427986 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.428350 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.428948 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.429076 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.429175 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.429247 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:35:26.442072 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.438493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38913 (local address 127.0.98.3:39707)
0927 12:35:26.438657 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.438662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.438821 (+ 159us) server_negotiation.cc:407] Connection header received
0927 12:35:26.438938 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.438940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.439012 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.439078 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:26.439338 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.439827 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.441051 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.441254 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.441324 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.441671 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.441740 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.441788 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.441887 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.441934 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:35:26.442124 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.438362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39707 (local address 127.0.98.2:38913)
0927 12:35:26.438705 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.438721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.438839 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.439102 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.439105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.439112 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:26.439249 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.439255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.439916 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.439919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.440942 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.440954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.441351 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.441355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.441451 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.441519 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.441806 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.441812 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.441871 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.441955 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":114,"threads_started":1}
I20260927 12:35:26.449503 2282 raft_consensus.cc:1261] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Refusing update from remote peer fa705e98965743558b77d9b74dc3e9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:26.449663 2562 raft_consensus.cc:1261] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Refusing update from remote peer fa705e98965743558b77d9b74dc3e9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:26.449945 2642 consensus_queue.cc:1035] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:26.450515 2642 consensus_queue.cc:1035] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:26.453840 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333939308332331008 to 7333939307413409792. Current Snapshot: MvccSnapshot[applied={T|T < 7333939308332331008}]
I20260927 12:35:26.473443 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.469673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43501 (local address 127.0.98.3:42359)
0927 12:35:26.469973 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:26.469988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:26.470093 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:26.470307 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:26.470310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:26.470318 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:26.470435 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.470440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.471228 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.471232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.472172 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:26.472183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.472681 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.472684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:26.472786 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.472844 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.473162 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.473168 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:35:26.473228 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:35:26.473312 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260927 12:35:26.473443 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:26.469731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42359 (local address 127.0.98.2:43501)
0927 12:35:26.469859 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:26.469864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:26.470042 (+ 178us) server_negotiation.cc:407] Connection header received
0927 12:35:26.470172 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:26.470174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:26.470216 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:26.470277 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:26.470570 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.471125 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.472323 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:26.472581 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:26.472678 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:26.472999 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:26.473081 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:26.473141 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:26.473238 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 12:35:26.473291 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:35:26.501065 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:26.501274 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:26.504240 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Sending DeleteTablet for 3 replicas of tablet e259ec06373c4d26a305b0fc04298ad5
I20260927 12:35:26.505097 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet e259ec06373c4d26a305b0fc04298ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260927 12:35:26.505347 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet e259ec06373c4d26a305b0fc04298ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49970
I20260927 12:35:26.505689 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet e259ec06373c4d26a305b0fc04298ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32816
I20260927 12:35:26.506280 2684 tablet_replica.cc:330] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: stopping tablet replica
I20260927 12:35:26.506290 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:26.506773 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:26.507104 2684 raft_consensus.cc:2229] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:26.507627 2684 raft_consensus.cc:2258] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:26.509307 2684 ts_tablet_manager.cc:1905] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:26.509653 2682 tablet_replica.cc:330] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: stopping tablet replica
I20260927 12:35:26.510046 2682 raft_consensus.cc:2229] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:26.510329 2682 raft_consensus.cc:2258] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:26.512594 2683 tablet_replica.cc:330] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: stopping tablet replica
I20260927 12:35:26.512651 2542 tablet_service.cc:1467] Processing CreateTablet for tablet e540ad81c6fe445681b7c2483fb89a6c (DEFAULT_TABLE table=test-table [id=86cf91834f964f4883aaad4ad1919ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:26.512761 2402 tablet_service.cc:1467] Processing CreateTablet for tablet e540ad81c6fe445681b7c2483fb89a6c (DEFAULT_TABLE table=test-table [id=86cf91834f964f4883aaad4ad1919ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:26.512948 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e540ad81c6fe445681b7c2483fb89a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:26.513056 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e540ad81c6fe445681b7c2483fb89a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:26.513214 2683 raft_consensus.cc:2229] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:26.513433 2683 raft_consensus.cc:2258] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:26.515247 2684 ts_tablet_manager.cc:1918] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:26.515455 2632 tablet_bootstrap.cc:492] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Bootstrap starting.
I20260927 12:35:26.515609 2684 log.cc:1196] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/e259ec06373c4d26a305b0fc04298ad5
I20260927 12:35:26.516007 2684 ts_tablet_manager.cc:1939] T e259ec06373c4d26a305b0fc04298ad5 P fa705e98965743558b77d9b74dc3e9e2: Deleting consensus metadata
I20260927 12:35:26.516265 2633 tablet_bootstrap.cc:492] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Bootstrap starting.
I20260927 12:35:26.516441 2632 tablet_bootstrap.cc:654] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:26.517175 2633 tablet_bootstrap.cc:654] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:26.518451 2112 catalog_manager.cc:4841] TS fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501): tablet e259ec06373c4d26a305b0fc04298ad5 (table test-table [id=f8abc440f10349de823b60f3dc42c00a]) successfully deleted
I20260927 12:35:26.519030 2633 tablet_bootstrap.cc:492] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: No bootstrap required, opened a new log
I20260927 12:35:26.519124 2633 ts_tablet_manager.cc:1397] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:26.519098 2262 tablet_service.cc:1467] Processing CreateTablet for tablet e540ad81c6fe445681b7c2483fb89a6c (DEFAULT_TABLE table=test-table [id=86cf91834f964f4883aaad4ad1919ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:26.519356 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e540ad81c6fe445681b7c2483fb89a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:26.519630 2633 raft_consensus.cc:348] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.519752 2633 raft_consensus.cc:374] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:26.519788 2633 raft_consensus.cc:729] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Initialized, Role: FOLLOWER
I20260927 12:35:26.519934 2633 consensus_queue.cc:260] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.520251 2633 ts_tablet_manager.cc:1428] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:26.522040 2683 ts_tablet_manager.cc:1905] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:26.522420 2631 tablet_bootstrap.cc:492] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Bootstrap starting.
I20260927 12:35:26.523370 2631 tablet_bootstrap.cc:654] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:26.525652 2682 ts_tablet_manager.cc:1905] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:26.525938 2632 tablet_bootstrap.cc:492] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: No bootstrap required, opened a new log
I20260927 12:35:26.526017 2632 ts_tablet_manager.cc:1397] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260927 12:35:26.526249 2683 ts_tablet_manager.cc:1918] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:26.526329 2683 log.cc:1196] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/e259ec06373c4d26a305b0fc04298ad5
I20260927 12:35:26.526502 2632 raft_consensus.cc:348] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.526607 2632 raft_consensus.cc:374] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:26.526636 2632 raft_consensus.cc:729] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8640c3820bee400e90b93792fae16472, State: Initialized, Role: FOLLOWER
I20260927 12:35:26.526660 2683 ts_tablet_manager.cc:1939] T e259ec06373c4d26a305b0fc04298ad5 P 8640c3820bee400e90b93792fae16472: Deleting consensus metadata
I20260927 12:35:26.526752 2632 consensus_queue.cc:260] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.527014 2632 ts_tablet_manager.cc:1428] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:26.528676 2111 catalog_manager.cc:4841] TS 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707): tablet e259ec06373c4d26a305b0fc04298ad5 (table test-table [id=f8abc440f10349de823b60f3dc42c00a]) successfully deleted
I20260927 12:35:26.530639 2682 ts_tablet_manager.cc:1918] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:26.530963 2682 log.cc:1196] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/e259ec06373c4d26a305b0fc04298ad5
I20260927 12:35:26.531289 2682 ts_tablet_manager.cc:1939] T e259ec06373c4d26a305b0fc04298ad5 P 26586d97a37240f083da6bead804f78f: Deleting consensus metadata
I20260927 12:35:26.532716 2110 catalog_manager.cc:4841] TS 26586d97a37240f083da6bead804f78f (127.0.98.1:33019): tablet e259ec06373c4d26a305b0fc04298ad5 (table test-table [id=f8abc440f10349de823b60f3dc42c00a]) successfully deleted
I20260927 12:35:26.533234 2631 tablet_bootstrap.cc:492] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: No bootstrap required, opened a new log
I20260927 12:35:26.533313 2631 ts_tablet_manager.cc:1397] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260927 12:35:26.533797 2631 raft_consensus.cc:348] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.533910 2631 raft_consensus.cc:374] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:26.533937 2631 raft_consensus.cc:729] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26586d97a37240f083da6bead804f78f, State: Initialized, Role: FOLLOWER
I20260927 12:35:26.534073 2631 consensus_queue.cc:260] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.534433 2631 ts_tablet_manager.cc:1428] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:35:26.568737 2637 raft_consensus.cc:482] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:26.568881 2637 raft_consensus.cc:504] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.569304 2637 leader_election.cc:288] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:26.569702 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540ad81c6fe445681b7c2483fb89a6c" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f" is_pre_election: true
I20260927 12:35:26.569767 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540ad81c6fe445681b7c2483fb89a6c" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472" is_pre_election: true
I20260927 12:35:26.569849 2282 raft_consensus.cc:2454] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 0.
I20260927 12:35:26.569908 2562 raft_consensus.cc:2454] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 0.
I20260927 12:35:26.570173 2354 leader_election.cc:302] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:26.570531 2637 raft_consensus.cc:2781] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:26.570600 2637 raft_consensus.cc:482] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:26.570628 2637 raft_consensus.cc:3037] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.571738 2637 raft_consensus.cc:504] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.572147 2637 leader_election.cc:288] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 election: Requested vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:26.572564 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540ad81c6fe445681b7c2483fb89a6c" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472"
I20260927 12:35:26.572585 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e540ad81c6fe445681b7c2483fb89a6c" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f"
I20260927 12:35:26.572700 2562 raft_consensus.cc:3037] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.572701 2282 raft_consensus.cc:3037] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:26.573629 2282 raft_consensus.cc:2454] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 1.
I20260927 12:35:26.573876 2562 raft_consensus.cc:2454] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 1.
I20260927 12:35:26.573938 2354 leader_election.cc:302] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:26.574105 2637 raft_consensus.cc:2781] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:26.574186 2637 raft_consensus.cc:686] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 1 LEADER]: Becoming Leader. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Running, Role: LEADER
I20260927 12:35:26.574343 2637 consensus_queue.cc:237] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:26.576059 2120 catalog_manager.cc:5495] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa705e98965743558b77d9b74dc3e9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:26.595825 2562 raft_consensus.cc:1261] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Refusing update from remote peer fa705e98965743558b77d9b74dc3e9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:26.596338 2677 consensus_queue.cc:1035] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:26.596475 2282 raft_consensus.cc:1261] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Refusing update from remote peer fa705e98965743558b77d9b74dc3e9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:26.596899 2637 consensus_queue.cc:1035] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:35:26.612056 2663 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 12:35:28.164975 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:28.165144 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:28.167591 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Sending DeleteTablet for 3 replicas of tablet e540ad81c6fe445681b7c2483fb89a6c
I20260927 12:35:28.168232 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet e540ad81c6fe445681b7c2483fb89a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32816
I20260927 12:35:28.168246 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet e540ad81c6fe445681b7c2483fb89a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260927 12:35:28.168292 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet e540ad81c6fe445681b7c2483fb89a6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49970
I20260927 12:35:28.168641 2714 tablet_replica.cc:330] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: stopping tablet replica
I20260927 12:35:28.168778 2714 raft_consensus.cc:2229] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:28.168895 2714 raft_consensus.cc:2258] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:28.169119 2716 tablet_replica.cc:330] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: stopping tablet replica
I20260927 12:35:28.169232 2716 raft_consensus.cc:2229] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:28.169442 2716 raft_consensus.cc:2258] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:28.169505 2714 ts_tablet_manager.cc:1905] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:28.169691 2715 tablet_replica.cc:330] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: stopping tablet replica
I20260927 12:35:28.169808 2715 raft_consensus.cc:2229] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:28.169929 2715 raft_consensus.cc:2258] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:28.170413 2716 ts_tablet_manager.cc:1905] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:28.171691 2715 ts_tablet_manager.cc:1905] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:28.172209 2714 ts_tablet_manager.cc:1918] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:35:28.172279 2714 log.cc:1196] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/e540ad81c6fe445681b7c2483fb89a6c
I20260927 12:35:28.172495 2714 ts_tablet_manager.cc:1939] T e540ad81c6fe445681b7c2483fb89a6c P 26586d97a37240f083da6bead804f78f: Deleting consensus metadata
I20260927 12:35:28.172662 2716 ts_tablet_manager.cc:1918] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:35:28.172727 2716 log.cc:1196] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/e540ad81c6fe445681b7c2483fb89a6c
I20260927 12:35:28.172966 2716 ts_tablet_manager.cc:1939] T e540ad81c6fe445681b7c2483fb89a6c P fa705e98965743558b77d9b74dc3e9e2: Deleting consensus metadata
I20260927 12:35:28.173189 2110 catalog_manager.cc:4841] TS 26586d97a37240f083da6bead804f78f (127.0.98.1:33019): tablet e540ad81c6fe445681b7c2483fb89a6c (table test-table [id=86cf91834f964f4883aaad4ad1919ca0]) successfully deleted
I20260927 12:35:28.173548 2112 catalog_manager.cc:4841] TS fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501): tablet e540ad81c6fe445681b7c2483fb89a6c (table test-table [id=86cf91834f964f4883aaad4ad1919ca0]) successfully deleted
I20260927 12:35:28.174006 2715 ts_tablet_manager.cc:1918] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:35:28.174064 2715 log.cc:1196] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/e540ad81c6fe445681b7c2483fb89a6c
I20260927 12:35:28.174217 2715 ts_tablet_manager.cc:1939] T e540ad81c6fe445681b7c2483fb89a6c P 8640c3820bee400e90b93792fae16472: Deleting consensus metadata
I20260927 12:35:28.174885 2111 catalog_manager.cc:4841] TS 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707): tablet e540ad81c6fe445681b7c2483fb89a6c (table test-table [id=86cf91834f964f4883aaad4ad1919ca0]) successfully deleted
Sep 27 12:35:28 dist-test-slave-tzgx krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512528, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:35:28 dist-test-slave-tzgx krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512528, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:28.197639 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.189231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.0.98.62:45363)
0927 12:35:28.189518 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.189526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.189542 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:28.189651 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.189654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.189706 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.189781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:28.190510 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.191062 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.191872 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.192216 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.192315 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.194694 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:28.194698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:28.194701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:28.194730 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:28.196119 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:28.196498 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:28.196502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:28.196504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:28.196582 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:28.196816 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:28.196819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:28.196821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:28.197004 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:28.197123 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.197344 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.197472 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260927 12:35:28.200344 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:28.200789 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:28.205032 2262 tablet_service.cc:1467] Processing CreateTablet for tablet cd6c7151b0624d2cbfc52b58af39e4ec (DEFAULT_TABLE table=test-table [id=9bd13440b001457a955c6dfd64a955dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:28.205154 2402 tablet_service.cc:1467] Processing CreateTablet for tablet cd6c7151b0624d2cbfc52b58af39e4ec (DEFAULT_TABLE table=test-table [id=9bd13440b001457a955c6dfd64a955dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:28.205201 2542 tablet_service.cc:1467] Processing CreateTablet for tablet cd6c7151b0624d2cbfc52b58af39e4ec (DEFAULT_TABLE table=test-table [id=9bd13440b001457a955c6dfd64a955dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:28.205328 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6c7151b0624d2cbfc52b58af39e4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:28.205361 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6c7151b0624d2cbfc52b58af39e4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:28.205490 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6c7151b0624d2cbfc52b58af39e4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:28.207756 2725 tablet_bootstrap.cc:492] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Bootstrap starting.
I20260927 12:35:28.208240 2726 tablet_bootstrap.cc:492] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Bootstrap starting.
I20260927 12:35:28.208642 2725 tablet_bootstrap.cc:654] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:28.209044 2726 tablet_bootstrap.cc:654] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:28.209652 2727 tablet_bootstrap.cc:492] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Bootstrap starting.
I20260927 12:35:28.209805 2725 tablet_bootstrap.cc:492] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: No bootstrap required, opened a new log
I20260927 12:35:28.209892 2725 ts_tablet_manager.cc:1397] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:28.210083 2726 tablet_bootstrap.cc:492] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: No bootstrap required, opened a new log
I20260927 12:35:28.210161 2726 ts_tablet_manager.cc:1397] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:28.210304 2725 raft_consensus.cc:348] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.210433 2727 tablet_bootstrap.cc:654] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:28.210431 2725 raft_consensus.cc:374] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:28.210467 2725 raft_consensus.cc:729] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8640c3820bee400e90b93792fae16472, State: Initialized, Role: FOLLOWER
I20260927 12:35:28.210608 2725 consensus_queue.cc:260] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.210690 2726 raft_consensus.cc:348] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.210799 2726 raft_consensus.cc:374] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:28.210830 2726 raft_consensus.cc:729] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Initialized, Role: FOLLOWER
I20260927 12:35:28.210968 2726 consensus_queue.cc:260] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.211238 2726 ts_tablet_manager.cc:1428] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:28.211316 2725 ts_tablet_manager.cc:1428] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:28.211545 2727 tablet_bootstrap.cc:492] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: No bootstrap required, opened a new log
I20260927 12:35:28.211619 2727 ts_tablet_manager.cc:1397] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:28.212149 2727 raft_consensus.cc:348] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.212239 2727 raft_consensus.cc:374] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:28.212260 2727 raft_consensus.cc:729] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26586d97a37240f083da6bead804f78f, State: Initialized, Role: FOLLOWER
I20260927 12:35:28.212352 2727 consensus_queue.cc:260] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.212659 2727 ts_tablet_manager.cc:1428] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:28.298614 2731 raft_consensus.cc:482] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:28.298753 2731 raft_consensus.cc:504] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.299860 2731 leader_election.cc:288] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:28.305127 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.300261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52605 (local address 127.0.98.3:39707)
0927 12:35:28.300562 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.300571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.300686 (+ 115us) server_negotiation.cc:407] Connection header received
0927 12:35:28.300829 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.300832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.300882 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.300991 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:28.301364 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.302183 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.304069 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.304544 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.304650 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.304715 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.304915 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.304994 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":119,"threads_started":1}
I20260927 12:35:28.305132 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.300269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55789 (local address 127.0.98.2:43501)
0927 12:35:28.300562 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.300571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.300660 (+ 89us) server_negotiation.cc:407] Connection header received
0927 12:35:28.300829 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.300832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.300882 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.300991 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:28.301364 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.302195 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.304059 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.304518 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.304606 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.304673 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.304903 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.304984 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":120,"threads_started":1}
I20260927 12:35:28.305140 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.300163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43501 (local address 127.0.98.1:55789)
0927 12:35:28.300566 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:28.300581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:28.300731 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:28.301011 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:28.301014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:28.301042 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:28.301246 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.301253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.302325 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.302329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.303915 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.303929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.304652 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.304655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.304790 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.304814 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:35:28.304875 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:35:28.304956 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":138,"spinlock_wait_cycles":19456,"thread_start_us":128,"threads_started":1}
I20260927 12:35:28.305151 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.300163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39707 (local address 127.0.98.1:52605)
0927 12:35:28.300615 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:28.300625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:28.300730 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:28.301011 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:28.301014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:28.301041 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:28.301246 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.301253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.302325 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.302329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.303938 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.303945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.304666 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.304670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.304799 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.304822 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:35:28.304889 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:35:28.304965 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":278,"spinlock_wait_cycles":26880,"thread_start_us":87,"threads_started":1}
I20260927 12:35:28.305591 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6c7151b0624d2cbfc52b58af39e4ec" candidate_uuid: "26586d97a37240f083da6bead804f78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472" is_pre_election: true
I20260927 12:35:28.305745 2562 raft_consensus.cc:2454] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26586d97a37240f083da6bead804f78f in term 0.
I20260927 12:35:28.305943 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6c7151b0624d2cbfc52b58af39e4ec" candidate_uuid: "26586d97a37240f083da6bead804f78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa705e98965743558b77d9b74dc3e9e2" is_pre_election: true
I20260927 12:35:28.306078 2422 raft_consensus.cc:2454] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26586d97a37240f083da6bead804f78f in term 0.
I20260927 12:35:28.306138 2215 leader_election.cc:302] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, 8640c3820bee400e90b93792fae16472; no voters:
I20260927 12:35:28.306399 2731 raft_consensus.cc:2781] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:28.306453 2731 raft_consensus.cc:482] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:28.306496 2731 raft_consensus.cc:3037] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:28.307499 2731 raft_consensus.cc:504] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.307827 2731 leader_election.cc:288] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [CANDIDATE]: Term 1 election: Requested vote from peers fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:28.308297 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6c7151b0624d2cbfc52b58af39e4ec" candidate_uuid: "26586d97a37240f083da6bead804f78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa705e98965743558b77d9b74dc3e9e2"
I20260927 12:35:28.308297 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6c7151b0624d2cbfc52b58af39e4ec" candidate_uuid: "26586d97a37240f083da6bead804f78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472"
I20260927 12:35:28.308432 2562 raft_consensus.cc:3037] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:28.308432 2422 raft_consensus.cc:3037] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:28.309612 2422 raft_consensus.cc:2454] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26586d97a37240f083da6bead804f78f in term 1.
I20260927 12:35:28.309612 2562 raft_consensus.cc:2454] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26586d97a37240f083da6bead804f78f in term 1.
I20260927 12:35:28.309916 2216 leader_election.cc:302] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:28.310101 2731 raft_consensus.cc:2781] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:28.310307 2731 raft_consensus.cc:686] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 1 LEADER]: Becoming Leader. State: Replica: 26586d97a37240f083da6bead804f78f, State: Running, Role: LEADER
I20260927 12:35:28.310562 2731 consensus_queue.cc:237] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.312867 2123 catalog_manager.cc:5495] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26586d97a37240f083da6bead804f78f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "26586d97a37240f083da6bead804f78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:28.340801 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.337240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.0.98.1:33019)
0927 12:35:28.337498 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.337505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.337523 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:35:28.337589 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.337591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.337640 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.337744 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:28.337986 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.338519 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.339345 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.339580 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.339664 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.339929 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:28.340413 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:28.340523 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.340588 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.340643 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260927 12:35:28.351046 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.347841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33935 (local address 127.0.98.3:39707)
0927 12:35:28.348012 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.348017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.348115 (+ 98us) server_negotiation.cc:407] Connection header received
0927 12:35:28.348266 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.348269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.348314 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.348376 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:28.348639 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.349143 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.350023 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.350235 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.350322 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.350612 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:28.350685 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:28.350743 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.350838 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.350891 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:35:28.351056 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.347711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39707 (local address 127.0.98.1:33935)
0927 12:35:28.348014 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:28.348028 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:28.348164 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:28.348396 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:28.348399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:28.348407 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:28.348532 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.348538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.349245 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.349248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.349903 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.349911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.350347 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.350351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.350436 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.350478 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:28.350765 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:28.350771 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:35:28.350828 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:35:28.350912 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260927 12:35:28.356588 2422 raft_consensus.cc:1261] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Refusing update from remote peer 26586d97a37240f083da6bead804f78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:28.356614 2562 raft_consensus.cc:1261] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Refusing update from remote peer 26586d97a37240f083da6bead804f78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:28.357003 2736 consensus_queue.cc:1035] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:28.357184 2731 consensus_queue.cc:1035] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:28.373605 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.370287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33019 (local address 127.0.98.3:34435)
0927 12:35:28.370577 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:28.370590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:28.370695 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:28.370913 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:28.370915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:28.370921 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:28.371010 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.371016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.371701 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.371704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.372405 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:28.372415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.372916 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.372920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:28.372997 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.373024 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:28.373281 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:28.373287 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:35:28.373343 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:35:28.373422 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260927 12:35:28.373596 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.370343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34435 (local address 127.0.98.1:33019)
0927 12:35:28.370456 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.370461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.370672 (+ 211us) server_negotiation.cc:407] Connection header received
0927 12:35:28.370779 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.370782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.370839 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.370898 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:28.371075 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.371617 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.372500 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.372796 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.372908 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.373133 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:28.373218 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:28.373280 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.373441 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.373496 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 12:35:28.395684 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:28.395830 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:28.398092 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Sending DeleteTablet for 3 replicas of tablet cd6c7151b0624d2cbfc52b58af39e4ec
I20260927 12:35:28.398819 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6c7151b0624d2cbfc52b58af39e4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32816
I20260927 12:35:28.398882 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6c7151b0624d2cbfc52b58af39e4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49970
I20260927 12:35:28.399024 2716 tablet_replica.cc:330] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: stopping tablet replica
I20260927 12:35:28.399137 2716 raft_consensus.cc:2229] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:28.399260 2716 raft_consensus.cc:2258] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:28.399453 2715 tablet_replica.cc:330] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: stopping tablet replica
I20260927 12:35:28.399473 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6c7151b0624d2cbfc52b58af39e4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260927 12:35:28.399570 2715 raft_consensus.cc:2229] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:28.399648 2714 tablet_replica.cc:330] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: stopping tablet replica
I20260927 12:35:28.399551 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:35:28.399677 2715 raft_consensus.cc:2258] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:28.399744 2714 raft_consensus.cc:2229] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 1 LEADER]: Raft consensus shutting down.
W20260927 12:35:28.400005 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:28.400070 2714 raft_consensus.cc:2258] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:28.400746 2716 ts_tablet_manager.cc:1905] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:28.401129 2714 ts_tablet_manager.cc:1905] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:28.403448 2716 ts_tablet_manager.cc:1918] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:28.403532 2716 log.cc:1196] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/cd6c7151b0624d2cbfc52b58af39e4ec
I20260927 12:35:28.403571 2714 ts_tablet_manager.cc:1918] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:28.403640 2714 log.cc:1196] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/cd6c7151b0624d2cbfc52b58af39e4ec
I20260927 12:35:28.403766 2716 ts_tablet_manager.cc:1939] T cd6c7151b0624d2cbfc52b58af39e4ec P fa705e98965743558b77d9b74dc3e9e2: Deleting consensus metadata
I20260927 12:35:28.403892 2714 ts_tablet_manager.cc:1939] T cd6c7151b0624d2cbfc52b58af39e4ec P 26586d97a37240f083da6bead804f78f: Deleting consensus metadata
I20260927 12:35:28.404536 2112 catalog_manager.cc:4841] TS fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501): tablet cd6c7151b0624d2cbfc52b58af39e4ec (table test-table [id=9bd13440b001457a955c6dfd64a955dc]) successfully deleted
I20260927 12:35:28.405098 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 076e74b40b164b73bd09419cb0693b62 (DEFAULT_TABLE table=test-table [id=dae341f61d3e43edbe0dfb8a99375751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:28.405296 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 076e74b40b164b73bd09419cb0693b62 (DEFAULT_TABLE table=test-table [id=dae341f61d3e43edbe0dfb8a99375751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:28.405354 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076e74b40b164b73bd09419cb0693b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:28.405529 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076e74b40b164b73bd09419cb0693b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:28.406312 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 076e74b40b164b73bd09419cb0693b62 (DEFAULT_TABLE table=test-table [id=dae341f61d3e43edbe0dfb8a99375751]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:28.406586 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076e74b40b164b73bd09419cb0693b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:28.407578 2726 tablet_bootstrap.cc:492] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Bootstrap starting.
I20260927 12:35:28.407660 2727 tablet_bootstrap.cc:492] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f: Bootstrap starting.
I20260927 12:35:28.408438 2726 tablet_bootstrap.cc:654] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:28.408556 2727 tablet_bootstrap.cc:654] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:28.408968 2725 tablet_bootstrap.cc:492] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Bootstrap starting.
I20260927 12:35:28.409771 2725 tablet_bootstrap.cc:654] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:28.410152 2110 catalog_manager.cc:4841] TS 26586d97a37240f083da6bead804f78f (127.0.98.1:33019): tablet cd6c7151b0624d2cbfc52b58af39e4ec (table test-table [id=9bd13440b001457a955c6dfd64a955dc]) successfully deleted
I20260927 12:35:28.410216 2727 tablet_bootstrap.cc:492] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f: No bootstrap required, opened a new log
I20260927 12:35:28.410288 2727 ts_tablet_manager.cc:1397] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:28.410749 2727 raft_consensus.cc:348] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.410866 2727 raft_consensus.cc:374] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:28.410908 2727 raft_consensus.cc:729] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26586d97a37240f083da6bead804f78f, State: Initialized, Role: FOLLOWER
I20260927 12:35:28.410979 2715 ts_tablet_manager.cc:1905] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:28.411026 2727 consensus_queue.cc:260] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.411077 2725 tablet_bootstrap.cc:492] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: No bootstrap required, opened a new log
I20260927 12:35:28.411147 2725 ts_tablet_manager.cc:1397] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:28.411280 2727 ts_tablet_manager.cc:1428] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:28.411515 2725 raft_consensus.cc:348] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.411636 2725 raft_consensus.cc:374] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:28.411667 2725 raft_consensus.cc:729] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8640c3820bee400e90b93792fae16472, State: Initialized, Role: FOLLOWER
I20260927 12:35:28.411777 2725 consensus_queue.cc:260] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.412035 2725 ts_tablet_manager.cc:1428] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:28.413297 2715 ts_tablet_manager.cc:1918] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:28.413374 2715 log.cc:1196] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/cd6c7151b0624d2cbfc52b58af39e4ec
I20260927 12:35:28.413440 2726 tablet_bootstrap.cc:492] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: No bootstrap required, opened a new log
I20260927 12:35:28.413523 2726 ts_tablet_manager.cc:1397] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 12:35:28.413595 2715 ts_tablet_manager.cc:1939] T cd6c7151b0624d2cbfc52b58af39e4ec P 8640c3820bee400e90b93792fae16472: Deleting consensus metadata
I20260927 12:35:28.413956 2726 raft_consensus.cc:348] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.414067 2726 raft_consensus.cc:374] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:28.414101 2726 raft_consensus.cc:729] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Initialized, Role: FOLLOWER
I20260927 12:35:28.414175 2111 catalog_manager.cc:4841] TS 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707): tablet cd6c7151b0624d2cbfc52b58af39e4ec (table test-table [id=9bd13440b001457a955c6dfd64a955dc]) successfully deleted
I20260927 12:35:28.414223 2726 consensus_queue.cc:260] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.414481 2726 ts_tablet_manager.cc:1428] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:28.517882 2709 raft_consensus.cc:482] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:28.518056 2709 raft_consensus.cc:504] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.518604 2709 leader_election.cc:288] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:28.519114 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076e74b40b164b73bd09419cb0693b62" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472" is_pre_election: true
I20260927 12:35:28.519134 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076e74b40b164b73bd09419cb0693b62" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f" is_pre_election: true
I20260927 12:35:28.519286 2562 raft_consensus.cc:2454] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 0.
I20260927 12:35:28.519301 2281 raft_consensus.cc:2454] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 0.
I20260927 12:35:28.519722 2355 leader_election.cc:302] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26586d97a37240f083da6bead804f78f, 8640c3820bee400e90b93792fae16472, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:28.519948 2709 raft_consensus.cc:2781] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:28.520013 2709 raft_consensus.cc:482] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:28.520052 2709 raft_consensus.cc:3037] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:28.521430 2709 raft_consensus.cc:504] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.521848 2709 leader_election.cc:288] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 election: Requested vote from peers 26586d97a37240f083da6bead804f78f (127.0.98.1:33019), 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707)
I20260927 12:35:28.522244 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076e74b40b164b73bd09419cb0693b62" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26586d97a37240f083da6bead804f78f"
I20260927 12:35:28.522380 2281 raft_consensus.cc:3037] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:28.522389 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076e74b40b164b73bd09419cb0693b62" candidate_uuid: "fa705e98965743558b77d9b74dc3e9e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8640c3820bee400e90b93792fae16472"
I20260927 12:35:28.522513 2562 raft_consensus.cc:3037] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:28.523605 2281 raft_consensus.cc:2454] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 1.
I20260927 12:35:28.523605 2562 raft_consensus.cc:2454] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa705e98965743558b77d9b74dc3e9e2 in term 1.
I20260927 12:35:28.523933 2355 leader_election.cc:302] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8640c3820bee400e90b93792fae16472, fa705e98965743558b77d9b74dc3e9e2; no voters:
I20260927 12:35:28.524142 2709 raft_consensus.cc:2781] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:28.524319 2709 raft_consensus.cc:686] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 1 LEADER]: Becoming Leader. State: Replica: fa705e98965743558b77d9b74dc3e9e2, State: Running, Role: LEADER
I20260927 12:35:28.524458 2709 consensus_queue.cc:237] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } }
I20260927 12:35:28.526046 2120 catalog_manager.cc:5495] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa705e98965743558b77d9b74dc3e9e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa705e98965743558b77d9b74dc3e9e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:28.547816 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:28.543999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.0.98.2:43501)
0927 12:35:28.544142 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:28.544147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:28.544162 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:35:28.544226 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:28.544229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:28.544281 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:28.544350 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:28.544568 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.545359 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.546585 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:28.546889 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:28.546992 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:28.547339 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:28.547435 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:28.547505 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:28.547618 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 12:35:28.547680 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:35:28.554919 2562 raft_consensus.cc:1261] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Refusing update from remote peer fa705e98965743558b77d9b74dc3e9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:28.554975 2281 raft_consensus.cc:1261] T 076e74b40b164b73bd09419cb0693b62 P 26586d97a37240f083da6bead804f78f [term 1 FOLLOWER]: Refusing update from remote peer fa705e98965743558b77d9b74dc3e9e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:28.555382 2709 consensus_queue.cc:1035] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8640c3820bee400e90b93792fae16472" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:28.555547 2637 consensus_queue.cc:1035] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26586d97a37240f083da6bead804f78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:35:28.569337 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 12:35:30.119879 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:30.120048 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:30.122787 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dea29c1ea771429fa04d31ef06f193f2: Sending DeleteTablet for 3 replicas of tablet 076e74b40b164b73bd09419cb0693b62
I20260927 12:35:30.123425 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet 076e74b40b164b73bd09419cb0693b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260927 12:35:30.123484 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 076e74b40b164b73bd09419cb0693b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49970
I20260927 12:35:30.123498 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 076e74b40b164b73bd09419cb0693b62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32816
I20260927 12:35:30.123817 2778 tablet_replica.cc:330] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: stopping tablet replica
I20260927 12:35:30.123819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 2197
I20260927 12:35:30.124922 2778 raft_consensus.cc:2229] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:30.125137 2778 raft_consensus.cc:2258] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:30.125396 2779 tablet_replica.cc:330] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: stopping tablet replica
I20260927 12:35:30.125523 2779 raft_consensus.cc:2229] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:30.125645 2779 raft_consensus.cc:2258] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:30.125774 2778 ts_tablet_manager.cc:1905] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:30.126636 2779 ts_tablet_manager.cc:1905] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:30.128779 2778 ts_tablet_manager.cc:1918] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:35:30.128870 2778 log.cc:1196] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/076e74b40b164b73bd09419cb0693b62
I20260927 12:35:30.129133 2778 ts_tablet_manager.cc:1939] T 076e74b40b164b73bd09419cb0693b62 P fa705e98965743558b77d9b74dc3e9e2: Deleting consensus metadata
I20260927 12:35:30.129295 2779 ts_tablet_manager.cc:1918] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:35:30.129371 2779 log.cc:1196] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/076e74b40b164b73bd09419cb0693b62
I20260927 12:35:30.129595 2779 ts_tablet_manager.cc:1939] T 076e74b40b164b73bd09419cb0693b62 P 8640c3820bee400e90b93792fae16472: Deleting consensus metadata
I20260927 12:35:30.130035 2112 catalog_manager.cc:4841] TS fa705e98965743558b77d9b74dc3e9e2 (127.0.98.2:43501): tablet 076e74b40b164b73bd09419cb0693b62 (table test-table [id=dae341f61d3e43edbe0dfb8a99375751]) successfully deleted
I20260927 12:35:30.130257 2111 catalog_manager.cc:4841] TS 8640c3820bee400e90b93792fae16472 (127.0.98.3:39707): tablet 076e74b40b164b73bd09419cb0693b62 (table test-table [id=dae341f61d3e43edbe0dfb8a99375751]) successfully deleted
W20260927 12:35:30.131989 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33019 (error 108)
I20260927 12:35:30.132779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 2337
I20260927 12:35:30.133821 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:30.132800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33019 (local address 127.0.0.1:44782)
0927 12:35:30.133633 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:30.133715 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":98,"threads_started":1}
W20260927 12:35:30.133931 2110 catalog_manager.cc:4584] TS 26586d97a37240f083da6bead804f78f (127.0.98.1:33019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 076e74b40b164b73bd09419cb0693b62: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33019: connect: Connection refused (error 111)
I20260927 12:35:30.139263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 2477
I20260927 12:35:30.145185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 2093
2026-09-27T12:35:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6377 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:30 dist-test-slave-tzgx krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:30 dist-test-slave-tzgx krb5kdc[2784](info): set up 2 sockets
Sep 27 12:35:30 dist-test-slave-tzgx krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260927 12:35:31.111832 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:31.235589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34363
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:31.347716 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:31.348011 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:31.348053 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:31.352270 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:31.352341 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:31.352360 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:31.352376 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:31.352391 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:31.358021 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:31.359120 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:31.360193 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:31.369272 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:31.369359 2797 server_base.cc:1034] running on GCE node
W20260927 12:35:31.369288 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:31.369290 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:31.370211 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:31.371434 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:31.372596 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790512531372584 us; error 55 us; skew 500 ppm
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:31.375514 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:31.377240 2797 webserver.cc:460] Webserver started at http://127.0.98.62:38651/ using document root <none> and password file <none>
I20260927 12:35:31.377943 2797 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:31.378026 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:31.378602 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:31.381035 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "81557a8e228b4bf9a6e090e529273db2"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "b58c5153be6541b375eae66f33445d51"
server_key_iv: "a285947742252e8270d668c77793f34d"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.381630 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "81557a8e228b4bf9a6e090e529273db2"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "b58c5153be6541b375eae66f33445d51"
server_key_iv: "a285947742252e8270d668c77793f34d"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.385751 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:31.388417 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.389868 2797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:35:31.390009 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "81557a8e228b4bf9a6e090e529273db2"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "b58c5153be6541b375eae66f33445d51"
server_key_iv: "a285947742252e8270d668c77793f34d"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.390132 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:31.417512 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:31.420818 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:31.421001 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:31.428022 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34363
I20260927 12:35:31.428123 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34363 every 8 connection(s)
I20260927 12:35:31.429086 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:31.432057 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:31.432355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 2797
I20260927 12:35:31.432448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:31.432700 392 external_mini_cluster.cc:1428] Setting key 9fa67b79944f6b995fc0cc45196e777b
I20260927 12:35:31.439589 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2: Bootstrap starting.
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:31.442353 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:31.443344 2865 log.cc:824] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:31.445751 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2: No bootstrap required, opened a new log
I20260927 12:35:31.448036 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:31.434006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.0.98.62:34363)
0927 12:35:31.434502 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:31.434522 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:31.434664 (+ 142us) server_negotiation.cc:407] Connection header received
0927 12:35:31.435479 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:31.435512 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:31.436011 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:31.436472 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:31.437119 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.438606 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.439493 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.440191 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.440354 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:31.443065 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:31.443072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:31.443095 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:31.443132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:31.444993 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:31.445536 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:31.445541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:31.445548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:31.445663 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:31.445964 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:31.445967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:31.445969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:31.446395 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:31.446549 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:31.446915 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 12:35:31.447231 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":130,"threads_started":1}
I20260927 12:35:31.449784 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81557a8e228b4bf9a6e090e529273db2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } }
I20260927 12:35:31.450073 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:31.450121 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81557a8e228b4bf9a6e090e529273db2, State: Initialized, Role: FOLLOWER
I20260927 12:35:31.450578 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81557a8e228b4bf9a6e090e529273db2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } }
I20260927 12:35:31.450712 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:31.450753 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:31.450824 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:31.451722 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81557a8e228b4bf9a6e090e529273db2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } }
I20260927 12:35:31.452039 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81557a8e228b4bf9a6e090e529273db2; no voters:
I20260927 12:35:31.452363 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:31.452482 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:31.452696 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [term 1 LEADER]: Becoming Leader. State: Replica: 81557a8e228b4bf9a6e090e529273db2, State: Running, Role: LEADER
I20260927 12:35:31.453076 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81557a8e228b4bf9a6e090e529273db2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } }
I20260927 12:35:31.453558 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:31.454620 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81557a8e228b4bf9a6e090e529273db2. Latest consensus state: current_term: 1 leader_uuid: "81557a8e228b4bf9a6e090e529273db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81557a8e228b4bf9a6e090e529273db2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } }
I20260927 12:35:31.454941 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81557a8e228b4bf9a6e090e529273db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81557a8e228b4bf9a6e090e529273db2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34363 } } }
I20260927 12:35:31.455070 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:31.455176 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2 [sys.catalog]: This master's current role is: LEADER
W20260927 12:35:31.458406 2885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:31.458490 2885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:31.458549 2884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:31.459262 2884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:31.464823 2884 catalog_manager.cc:1347] Generated new cluster ID: b1197f570fd44268baf3e8091911f7ce
I20260927 12:35:31.464910 2884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:31.508637 2884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:31.509961 2884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:31.523190 2884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81557a8e228b4bf9a6e090e529273db2: Generated new TSK 0
I20260927 12:35:31.524155 2884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:31.555878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34363
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:31.671767 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:31.672075 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:31.672160 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:31.676342 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:31.676416 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:31.676479 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:31.681945 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:31.683015 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:31.684020 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:31.693694 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:31.693688 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:31.693747 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:31.694089 2891 server_base.cc:1034] running on GCE node
I20260927 12:35:31.694527 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:31.695199 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:31.696354 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790512531696334 us; error 33 us; skew 500 ppm
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:31.699144 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:31.700021 2891 webserver.cc:460] Webserver started at http://127.0.98.1:40853/ using document root <none> and password file <none>
I20260927 12:35:31.700590 2891 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:31.700650 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:31.700814 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:31.702807 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "91a38219797f4ee19ab05230ddf9e983"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "f9d536bf1077ce9470a045e439bb9eef"
server_key_iv: "9f07c7faa145e960dd10bb6005cd53e7"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.703306 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91a38219797f4ee19ab05230ddf9e983"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "f9d536bf1077ce9470a045e439bb9eef"
server_key_iv: "9f07c7faa145e960dd10bb6005cd53e7"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.707310 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:35:31.709926 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.711191 2891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:35:31.711311 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "91a38219797f4ee19ab05230ddf9e983"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "f9d536bf1077ce9470a045e439bb9eef"
server_key_iv: "9f07c7faa145e960dd10bb6005cd53e7"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.711434 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:31.737409 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:31.740789 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:31.740983 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:31.741586 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:31.742424 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:31.742471 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.742524 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:31.742549 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.750952 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33749
I20260927 12:35:31.751041 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33749 every 8 connection(s)
I20260927 12:35:31.751966 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:31.752782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 2891
I20260927 12:35:31.752893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:31.753156 392 external_mini_cluster.cc:1428] Setting key d3ff1c953a5de4be5a8a6fce1391b4c5
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:35:31.767724 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:31.753944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55143 (local address 127.0.98.62:34363)
0927 12:35:31.754077 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:31.754081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:31.755247 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 12:35:31.757087 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:31.757091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:31.757148 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:31.757220 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:31.758447 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.759221 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.760554 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.760975 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.761105 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:31.764083 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:31.764106 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:31.764109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:31.764140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:31.765433 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:31.765989 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:31.765991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:31.765993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:31.766072 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:31.766466 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:31.766469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:31.766471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:31.766685 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:31.766765 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:31.767447 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 12:35:31.767572 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:35:31.768818 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:31.754535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34363 (local address 127.0.98.1:55143)
0927 12:35:31.755068 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:31.755132 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:31.756865 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:31.757420 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:31.757428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:31.757801 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:31.758296 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:31.758309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.759384 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.759388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:31.760424 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:31.760434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.761271 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.761276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:31.761472 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:31.762191 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:31.762210 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:31.763892 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:31.765578 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:31.765584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:31.765600 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:31.765891 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:31.766154 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:31.766156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:31.766171 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:31.766338 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:31.766787 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:31.766795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:31.767190 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 12:35:31.767406 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 12:35:31.767902 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":127,"threads_started":1}
I20260927 12:35:31.770040 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34363
I20260927 12:35:31.770293 3020 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:31.770800 3020 heartbeater.cc:511] Master 127.0.98.62:34363 requested a full tablet report, sending...
I20260927 12:35:31.772392 2829 ts_manager.cc:194] Registered new tserver with Master: 91a38219797f4ee19ab05230ddf9e983 (127.0.98.1:33749)
I20260927 12:35:31.773525 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:31.783653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34363
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:31.898183 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:31.898480 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:31.898541 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:31.902776 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:31.902856 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:31.902938 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:31.908484 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:31.909602 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:31.910658 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:31.921712 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:31.921685 3025 server_base.cc:1034] running on GCE node
W20260927 12:35:31.921675 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:31.922356 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:31.922889 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:31.924034 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:31.925231 3025 hybrid_clock.cc:648] HybridClock initialized: now 1790512531925192 us; error 53 us; skew 500 ppm
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:31.928241 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:31.929095 3025 webserver.cc:460] Webserver started at http://127.0.98.2:38479/ using document root <none> and password file <none>
I20260927 12:35:31.929641 3025 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:31.929704 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:31.929894 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:31.931927 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "41fe6b4afced4b9fa33a37f696173223"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "26de7ddf5850144a89762c805ad090d1"
server_key_iv: "228c4dda00f8ac0bf4f948f891e0c99e"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.932514 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "41fe6b4afced4b9fa33a37f696173223"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "26de7ddf5850144a89762c805ad090d1"
server_key_iv: "228c4dda00f8ac0bf4f948f891e0c99e"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.936388 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:31.938939 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.940284 3025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:31.940398 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "41fe6b4afced4b9fa33a37f696173223"
format_stamp: "Formatted at 2026-09-27 12:35:31 on dist-test-slave-tzgx"
server_key: "26de7ddf5850144a89762c805ad090d1"
server_key_iv: "228c4dda00f8ac0bf4f948f891e0c99e"
server_key_version: "encryptionkey@0"
I20260927 12:35:31.940516 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:31.965417 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:31.968775 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:31.968959 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:31.969568 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:31.970430 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:31.970484 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.970539 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:31.970563 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:31.979107 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37173
I20260927 12:35:31.979185 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37173 every 8 connection(s)
I20260927 12:35:31.980085 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 12:35:31 dist-test-slave-tzgx krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:31.990362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3025
I20260927 12:35:31.990460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:31.990695 392 external_mini_cluster.cc:1428] Setting key 0cf457f5727a3e60a35c06aa70fabafb
I20260927 12:35:31.994678 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:31.982093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52841 (local address 127.0.98.62:34363)
0927 12:35:31.982272 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:31.982277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:31.983223 (+ 946us) server_negotiation.cc:407] Connection header received
0927 12:35:31.984695 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:31.984698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:31.984750 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:31.984832 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:31.986097 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.986652 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.987831 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.988151 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.988265 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:31.990845 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:31.990849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:31.990851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:31.990877 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:31.992493 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:31.992974 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:31.992977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:31.992979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:31.993067 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:31.993429 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:31.993432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:31.993434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:31.993653 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:31.993749 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:31.994444 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 12:35:31.994555 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:35:31.995648 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:31.982503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34363 (local address 127.0.98.2:52841)
0927 12:35:31.983051 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:31.983104 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:31.984452 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:31.985027 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:31.985035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:31.985416 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:31.985930 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:31.985944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.986754 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.986757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:31.987684 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:31.987693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:31.988312 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:31.988315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:31.988477 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:31.989073 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:31.989089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:31.990664 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:31.992601 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:31.992606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:31.992620 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:31.992881 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:31.993154 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:31.993157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:31.993159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:31.993352 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:31.993757 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:31.993764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:31.994159 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 12:35:31.994386 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 12:35:31.994767 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260927 12:35:31.996819 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34363
I20260927 12:35:31.997058 3154 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:31.997568 3154 heartbeater.cc:511] Master 127.0.98.62:34363 requested a full tablet report, sending...
I20260927 12:35:31.998678 2829 ts_manager.cc:194] Registered new tserver with Master: 41fe6b4afced4b9fa33a37f696173223 (127.0.98.2:37173)
I20260927 12:35:31.999256 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52841
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:32.020002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34363
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:32.131755 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:32.132067 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:32.132150 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:32.136360 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:32.136427 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:32.136512 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:32.141897 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:32.143021 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:32.144078 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:35:32.153720 3159 server_base.cc:1034] running on GCE node
W20260927 12:35:32.153659 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:32.153672 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:32.153637 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:32.154387 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:32.155043 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:32.156188 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790512532156156 us; error 60 us; skew 500 ppm
Sep 27 12:35:32 dist-test-slave-tzgx krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:32.159006 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:32.159932 3159 webserver.cc:460] Webserver started at http://127.0.98.3:37877/ using document root <none> and password file <none>
I20260927 12:35:32.160506 3159 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:32.160571 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:32.160768 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:32.162855 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f5eded57a3942a28fa2a7a4792cdd0d"
format_stamp: "Formatted at 2026-09-27 12:35:32 on dist-test-slave-tzgx"
server_key: "58a237e997495bc1712ed197d2aa11a1"
server_key_iv: "aa9016ee32039098268835ea4af309c3"
server_key_version: "encryptionkey@0"
I20260927 12:35:32.163371 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f5eded57a3942a28fa2a7a4792cdd0d"
format_stamp: "Formatted at 2026-09-27 12:35:32 on dist-test-slave-tzgx"
server_key: "58a237e997495bc1712ed197d2aa11a1"
server_key_iv: "aa9016ee32039098268835ea4af309c3"
server_key_version: "encryptionkey@0"
I20260927 12:35:32.167068 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:35:32.169610 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:32.170887 3159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:32.171022 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "0f5eded57a3942a28fa2a7a4792cdd0d"
format_stamp: "Formatted at 2026-09-27 12:35:32 on dist-test-slave-tzgx"
server_key: "58a237e997495bc1712ed197d2aa11a1"
server_key_iv: "aa9016ee32039098268835ea4af309c3"
server_key_version: "encryptionkey@0"
I20260927 12:35:32.171124 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:32.185088 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:32.188171 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:32.188349 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:32.188937 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:32.189749 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:32.189795 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:32.189857 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:32.189883 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:32.198396 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38453
I20260927 12:35:32.198433 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38453 every 8 connection(s)
I20260927 12:35:32.199303 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:32.206552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3159
I20260927 12:35:32.206667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:32.207008 392 external_mini_cluster.cc:1428] Setting key 72881dc3bd6371eb5b04fbbdf8803b8b
Sep 27 12:35:32 dist-test-slave-tzgx krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:32.213919 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:32.201198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54465 (local address 127.0.98.62:34363)
0927 12:35:32.201374 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:32.201379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:32.202249 (+ 870us) server_negotiation.cc:407] Connection header received
0927 12:35:32.203554 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:32.203557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:32.203606 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:32.203689 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:32.205032 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:32.205597 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:32.206837 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:32.207155 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:32.207277 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:32.210270 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:32.210273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:32.210275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:32.210301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:32.211766 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:32.212355 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:32.212359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:32.212360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:32.212446 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:32.212790 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:32.212793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:32.212794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:32.213050 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:32.213138 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:32.213628 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 12:35:32.213762 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:35:32.214766 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:32.201619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34363 (local address 127.0.98.3:54465)
0927 12:35:32.202097 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:32.202140 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:32.203337 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:32.203886 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:32.203894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:32.204360 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:32.204859 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:32.204872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:32.205726 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:32.205729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:32.206665 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:32.206675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:32.207305 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:32.207308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:32.207501 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:32.208194 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:32.208211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:32.210024 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:32.211891 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:32.211897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:32.211912 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:32.212223 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:32.212525 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:32.212527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:32.212529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:32.212670 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:32.213136 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:32.213140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:32.213396 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 12:35:32.213553 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 12:35:32.213824 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260927 12:35:32.216020 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34363
I20260927 12:35:32.216275 3288 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:32.216778 3288 heartbeater.cc:511] Master 127.0.98.62:34363 requested a full tablet report, sending...
I20260927 12:35:32.217881 2829 ts_manager.cc:194] Registered new tserver with Master: 0f5eded57a3942a28fa2a7a4792cdd0d (127.0.98.3:38453)
I20260927 12:35:32.218437 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54465
I20260927 12:35:32.221204 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:35:32.234083 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:32.232542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.0.98.62:34363)
0927 12:35:32.232714 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:32.232719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:32.232910 (+ 191us) server_negotiation.cc:407] Connection header received
0927 12:35:32.233033 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:32.233037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:32.233084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:32.233154 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:32.233959 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40460: BlockingRecv error: recv got EOF from 127.0.0.1:40460 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:35:32.235193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 2891
I20260927 12:35:32.240036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3025
I20260927 12:35:32.245044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3159
I20260927 12:35:32.249892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 2797
2026-09-27T12:35:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2104 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:32 dist-test-slave-tzgx krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:32 dist-test-slave-tzgx krb5kdc[3300](info): set up 2 sockets
Sep 27 12:35:32 dist-test-slave-tzgx krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260927 12:35:33.220263 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:33 dist-test-slave-tzgx krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:33.339293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41711
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:33.451488 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:33.451797 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:33.451838 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:33.456067 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:33.456167 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:33.456185 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:33.456202 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:33.456215 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:33.461691 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:33.462765 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:33.463757 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:33.472172 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:33.472278 3313 server_base.cc:1034] running on GCE node
W20260927 12:35:33.472174 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:33.472172 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:33.472965 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:33.474074 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:33.475214 3313 hybrid_clock.cc:648] HybridClock initialized: now 1790512533475198 us; error 39 us; skew 500 ppm
Sep 27 12:35:33 dist-test-slave-tzgx krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:33.477962 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:33.478859 3313 webserver.cc:460] Webserver started at http://127.0.98.62:39107/ using document root <none> and password file <none>
I20260927 12:35:33.479399 3313 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:33.479465 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:33.479678 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:33.481751 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "11305fbdf38f40a9999b0b0b097b2327"
format_stamp: "Formatted at 2026-09-27 12:35:33 on dist-test-slave-tzgx"
server_key: "834f864b5a029a27b06ef41c85703e00"
server_key_iv: "ea2a1adb2d18c1d84133a5c23ba7aa5f"
server_key_version: "encryptionkey@0"
I20260927 12:35:33.482295 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "11305fbdf38f40a9999b0b0b097b2327"
format_stamp: "Formatted at 2026-09-27 12:35:33 on dist-test-slave-tzgx"
server_key: "834f864b5a029a27b06ef41c85703e00"
server_key_iv: "ea2a1adb2d18c1d84133a5c23ba7aa5f"
server_key_version: "encryptionkey@0"
I20260927 12:35:33.486178 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 12:35:33.488780 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:33.490255 3313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:33.490406 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "11305fbdf38f40a9999b0b0b097b2327"
format_stamp: "Formatted at 2026-09-27 12:35:33 on dist-test-slave-tzgx"
server_key: "834f864b5a029a27b06ef41c85703e00"
server_key_iv: "ea2a1adb2d18c1d84133a5c23ba7aa5f"
server_key_version: "encryptionkey@0"
I20260927 12:35:33.490520 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:33.507682 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:33.511034 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:33.511232 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:33.518384 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41711
I20260927 12:35:33.518693 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41711 every 8 connection(s)
I20260927 12:35:33.519380 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:33.522223 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:33.525496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3313
I20260927 12:35:33.525599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:33.525825 392 external_mini_cluster.cc:1428] Setting key a965ac617028b00d9a44de36af5a142a
I20260927 12:35:33.528041 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327: Bootstrap starting.
I20260927 12:35:33.530783 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:33.531651 3381 log.cc:824] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:33.533830 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327: No bootstrap required, opened a new log
Sep 27 12:35:33 dist-test-slave-tzgx krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:33.538005 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11305fbdf38f40a9999b0b0b097b2327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41711 } }
I20260927 12:35:33.538256 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:33.538316 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11305fbdf38f40a9999b0b0b097b2327, State: Initialized, Role: FOLLOWER
I20260927 12:35:33.538789 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11305fbdf38f40a9999b0b0b097b2327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41711 } }
I20260927 12:35:33.538923 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:33.538988 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:33.539075 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:33.540048 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11305fbdf38f40a9999b0b0b097b2327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41711 } }
I20260927 12:35:33.540422 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11305fbdf38f40a9999b0b0b097b2327; no voters:
I20260927 12:35:33.540570 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:33.527153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.0.98.62:41711)
0927 12:35:33.527571 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:33.527595 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:33.527636 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:35:33.528495 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:33.528518 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:33.528932 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:33.529306 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:33.529932 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:33.531393 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:33.532388 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:33.532875 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:33.532995 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:33.535809 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:33.535814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:33.535828 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:33.535889 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:33.537519 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:33.538014 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:33.538019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:33.538024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:33.538135 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:33.538453 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:33.538456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:33.538457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:33.538856 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:33.539049 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:33.539362 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 12:35:33.539665 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
I20260927 12:35:33.540715 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:33.540889 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:33.541220 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [term 1 LEADER]: Becoming Leader. State: Replica: 11305fbdf38f40a9999b0b0b097b2327, State: Running, Role: LEADER
I20260927 12:35:33.541615 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11305fbdf38f40a9999b0b0b097b2327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41711 } }
I20260927 12:35:33.541935 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:33.543166 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11305fbdf38f40a9999b0b0b097b2327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11305fbdf38f40a9999b0b0b097b2327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41711 } } }
I20260927 12:35:33.543291 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:33.543637 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11305fbdf38f40a9999b0b0b097b2327. Latest consensus state: current_term: 1 leader_uuid: "11305fbdf38f40a9999b0b0b097b2327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11305fbdf38f40a9999b0b0b097b2327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41711 } } }
I20260927 12:35:33.543733 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:33.544194 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:35:33.544018 3396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:33.547323 3396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:33.553530 3396 catalog_manager.cc:1347] Generated new cluster ID: 2d80cbe568334ed99a7e1b55f261bce6
I20260927 12:35:33.553607 3396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:33.566069 3396 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:33.567040 3396 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:33.570119 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:33.556784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.0.98.62:41711)
0927 12:35:33.564271 (+ 7487us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:33.564276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:33.564301 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:35:33.564355 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:33.564359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:33.564419 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:33.564540 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:33.565257 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:33.566066 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:33.567392 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:33.567780 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:33.568016 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:33.569011 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:33.569015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:33.569018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:33.569624 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:33.569661 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:33.569681 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:33.569764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:33.569884 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 12:35:33.569968 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7375}
I20260927 12:35:33.573163 3396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11305fbdf38f40a9999b0b0b097b2327: Generated new TSK 0
I20260927 12:35:33.574033 3396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:35:33.580554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3313
2026-09-27T12:35:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1328 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:33 dist-test-slave-tzgx krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:33 dist-test-slave-tzgx krb5kdc[3414](info): set up 2 sockets
Sep 27 12:35:33 dist-test-slave-tzgx krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260927 12:35:34.548552 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:34 dist-test-slave-tzgx krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:34.680382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:34.795066 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:34.795367 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:34.795409 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:34.799630 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:34.799710 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:34.799728 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:34.799744 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:34.799758 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:34.805370 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:34.806535 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:34.807592 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:34.816409 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:34.816543 3427 server_base.cc:1034] running on GCE node
W20260927 12:35:34.816412 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:34.816409 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:34.817309 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:34.818557 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:34.819764 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790512534819727 us; error 58 us; skew 500 ppm
Sep 27 12:35:34 dist-test-slave-tzgx krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:34.822932 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:34.823897 3427 webserver.cc:460] Webserver started at http://127.0.98.62:34989/ using document root <none> and password file <none>
I20260927 12:35:34.824501 3427 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:34.824568 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:34.824739 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:34.826881 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "7aeac0cba5354cc39d5b0dad5cff4476"
format_stamp: "Formatted at 2026-09-27 12:35:34 on dist-test-slave-tzgx"
server_key: "70142bab6598d7b12ece7b0850974e98"
server_key_iv: "5697dbf4143c390ca9af74cdd0081d06"
server_key_version: "encryptionkey@0"
I20260927 12:35:34.827436 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aeac0cba5354cc39d5b0dad5cff4476"
format_stamp: "Formatted at 2026-09-27 12:35:34 on dist-test-slave-tzgx"
server_key: "70142bab6598d7b12ece7b0850974e98"
server_key_iv: "5697dbf4143c390ca9af74cdd0081d06"
server_key_version: "encryptionkey@0"
I20260927 12:35:34.831331 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:34.834141 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:34.835616 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:34.835755 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "7aeac0cba5354cc39d5b0dad5cff4476"
format_stamp: "Formatted at 2026-09-27 12:35:34 on dist-test-slave-tzgx"
server_key: "70142bab6598d7b12ece7b0850974e98"
server_key_iv: "5697dbf4143c390ca9af74cdd0081d06"
server_key_version: "encryptionkey@0"
I20260927 12:35:34.835878 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:34.875810 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:34.879541 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:34.879709 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:34.886544 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37625
I20260927 12:35:34.886571 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37625 every 8 connection(s)
I20260927 12:35:34.887557 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:34.890389 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:34.896294 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476: Bootstrap starting.
I20260927 12:35:34.897516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3427
I20260927 12:35:34.897629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:34.897884 392 external_mini_cluster.cc:1428] Setting key 5a3e01814fb2fd9b04e451227abd64b2
I20260927 12:35:34.899272 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:34.900079 3495 log.cc:824] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:34.902295 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476: No bootstrap required, opened a new log
I20260927 12:35:34.906574 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
Sep 27 12:35:34 dist-test-slave-tzgx krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:34.906886 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:34.906981 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aeac0cba5354cc39d5b0dad5cff4476, State: Initialized, Role: FOLLOWER
I20260927 12:35:34.907492 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260927 12:35:34.907663 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:34.907734 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:34.907838 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:34.909030 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260927 12:35:34.909505 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aeac0cba5354cc39d5b0dad5cff4476; no voters:
I20260927 12:35:34.909901 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:34.909957 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:34.910135 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [term 1 LEADER]: Becoming Leader. State: Replica: 7aeac0cba5354cc39d5b0dad5cff4476, State: Running, Role: LEADER
I20260927 12:35:34.910502 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } }
I20260927 12:35:34.911237 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:34.912281 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeac0cba5354cc39d5b0dad5cff4476. Latest consensus state: current_term: 1 leader_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } } }
I20260927 12:35:34.912401 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:34.912652 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac0cba5354cc39d5b0dad5cff4476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37625 } } }
I20260927 12:35:34.912725 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:34.912779 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:34.913219 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:34.898982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.98.62:37625)
0927 12:35:34.899562 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:34.899578 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:34.899615 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:35:34.900380 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:34.900401 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:34.900749 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:34.901048 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:34.901896 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:34.902946 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:34.904134 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:34.904620 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:34.904747 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:34.907501 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:34.907507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:34.907526 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:34.907558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:34.909595 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:34.910130 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:34.910136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:34.910142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:34.910287 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:34.910599 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:34.910603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:34.910605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:34.911013 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:34.911193 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:34.911802 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 12:35:34.912171 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":139,"threads_started":1}
I20260927 12:35:34.916004 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:34.921604 3509 catalog_manager.cc:1347] Generated new cluster ID: cf6331ea53ab4464bcbb0d51c24276bb
I20260927 12:35:34.921681 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:34.944130 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:34.945125 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:34.956067 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aeac0cba5354cc39d5b0dad5cff4476: Generated new TSK 0
I20260927 12:35:34.956897 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:34.992635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:35.108248 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:35.108526 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:35.108572 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:35.112732 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:35.112802 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:35.112862 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:35.118311 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:35.119432 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:35.120472 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:35.130333 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:35.130339 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:35.130339 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:35.130398 3521 server_base.cc:1034] running on GCE node
I20260927 12:35:35.130987 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:35.131673 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:35.132845 3521 hybrid_clock.cc:648] HybridClock initialized: now 1790512535132823 us; error 34 us; skew 500 ppm
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:35.135689 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:35.136677 3521 webserver.cc:460] Webserver started at http://127.0.98.1:37761/ using document root <none> and password file <none>
I20260927 12:35:35.137229 3521 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:35.137290 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:35.137461 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:35.139730 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "320be3b29cc243caa1920e31e07eb77e"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "cf0fd95fd3e0e4fc70a18c77664816c2"
server_key_iv: "4c03da878e2e803e36fb2c60e9b23e7a"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.140444 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "320be3b29cc243caa1920e31e07eb77e"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "cf0fd95fd3e0e4fc70a18c77664816c2"
server_key_iv: "4c03da878e2e803e36fb2c60e9b23e7a"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.144785 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:35.147635 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.149302 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:35.149434 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "320be3b29cc243caa1920e31e07eb77e"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "cf0fd95fd3e0e4fc70a18c77664816c2"
server_key_iv: "4c03da878e2e803e36fb2c60e9b23e7a"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.149593 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:35.163784 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:35.166998 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:35.167172 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:35.167820 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:35.168731 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:35.168778 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.168819 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:35.168845 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.177490 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34923
I20260927 12:35:35.177568 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34923 every 8 connection(s)
I20260927 12:35:35.178478 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:35.179454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3521
I20260927 12:35:35.179553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:35.179821 392 external_mini_cluster.cc:1428] Setting key e525f375f9caced65a8ba65d4c623ce8
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:35.197814 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.180453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59907 (local address 127.0.98.62:37625)
0927 12:35:35.180579 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:35.180584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:35.181658 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 12:35:35.183232 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:35.183235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:35.183285 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:35.183370 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:35.185073 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.185685 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.187488 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.187762 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.187854 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.193677 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:35.193682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:35.193685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:35.193719 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:35.195310 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.196017 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.196020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.196021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.196148 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.196577 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.196580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.196581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.196780 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:35.196867 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:35.197573 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 12:35:35.197684 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:35.199185 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.180965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.1:59907)
0927 12:35:35.181511 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:35.181561 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:35.182956 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:35.183607 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:35.183618 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:35.184177 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:35.184890 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:35.184911 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.185830 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.185834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:35.187333 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:35.187348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.187939 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.187943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:35.188196 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.188881 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:35.188903 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:35.193434 (+ 4531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:35.195472 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:35.195479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:35.195500 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:35.195896 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:35.196244 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:35.196247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:35.196249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:35.196481 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:35.196883 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:35.196890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:35.197297 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 12:35:35.197557 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 12:35:35.197967 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":146,"threads_started":1}
I20260927 12:35:35.200601 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37625
I20260927 12:35:35.200845 3650 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:35.201352 3650 heartbeater.cc:511] Master 127.0.98.62:37625 requested a full tablet report, sending...
I20260927 12:35:35.203023 3459 ts_manager.cc:194] Registered new tserver with Master: 320be3b29cc243caa1920e31e07eb77e (127.0.98.1:34923)
I20260927 12:35:35.204322 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:35.211375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:35.323901 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:35.324203 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:35.324246 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:35.328553 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:35.328624 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:35.328684 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:35.334158 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:35.335253 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:35.336381 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:35.346724 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:35.346727 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:35.346727 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:35.347046 3655 server_base.cc:1034] running on GCE node
I20260927 12:35:35.347554 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:35.348299 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:35.349488 3655 hybrid_clock.cc:648] HybridClock initialized: now 1790512535349471 us; error 31 us; skew 500 ppm
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:35.355829 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:35.356853 3655 webserver.cc:460] Webserver started at http://127.0.98.2:40637/ using document root <none> and password file <none>
I20260927 12:35:35.357512 3655 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:35.357589 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:35.357800 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:35.359965 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "fee7673a4e3741fdb40a43b640a2d59d"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "373225f564ae6a26403b3a83ed24d688"
server_key_iv: "94e81d7503f0765b93c750c4d9bfc90e"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.360608 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fee7673a4e3741fdb40a43b640a2d59d"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "373225f564ae6a26403b3a83ed24d688"
server_key_iv: "94e81d7503f0765b93c750c4d9bfc90e"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.364603 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:35.367300 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.368880 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:35.369010 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "fee7673a4e3741fdb40a43b640a2d59d"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "373225f564ae6a26403b3a83ed24d688"
server_key_iv: "94e81d7503f0765b93c750c4d9bfc90e"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.369128 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:35.383695 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:35.387173 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:35.387368 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:35.388037 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:35.388929 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:35.388979 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.389036 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:35.389065 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.397917 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45589
I20260927 12:35:35.397957 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45589 every 8 connection(s)
I20260927 12:35:35.398819 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:35.408334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3655
I20260927 12:35:35.408437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:35.408751 392 external_mini_cluster.cc:1428] Setting key 1d180fdf4e84400c6a1110a9c70efca2
I20260927 12:35:35.413465 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.400686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60875 (local address 127.0.98.62:37625)
0927 12:35:35.400841 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:35.400844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:35.401730 (+ 886us) server_negotiation.cc:407] Connection header received
0927 12:35:35.403083 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:35.403086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:35.403140 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:35.403213 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:35.404474 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.405000 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.406125 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.406348 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.406437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.409164 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:35.409168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:35.409170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:35.409198 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:35.410782 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.411424 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.411427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.411429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.411530 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.411979 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.411983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.411985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.412330 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:35.412410 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:35.413182 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 12:35:35.413335 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:35:35.414772 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.401103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.2:60875)
0927 12:35:35.401575 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:35.401618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:35.402866 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:35.403398 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:35.403405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:35.403772 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:35.404327 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:35.404344 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.405117 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.405120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:35.406008 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:35.406017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.406489 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.406493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:35.406654 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.407209 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:35.407225 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:35.408933 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:35.410947 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:35.410952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:35.410967 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:35.411333 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:35.411613 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:35.411615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:35.411616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:35.411807 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:35.412497 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:35.412504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:35.412877 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 12:35:35.413631 (+ 754us) client_negotiation.cc:240] Negotiation successful
0927 12:35:35.413971 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260927 12:35:35.416080 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37625
I20260927 12:35:35.416368 3784 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:35.416883 3784 heartbeater.cc:511] Master 127.0.98.62:37625 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:35.418061 3459 ts_manager.cc:194] Registered new tserver with Master: fee7673a4e3741fdb40a43b640a2d59d (127.0.98.2:45589)
I20260927 12:35:35.418663 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:35.435938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:35.545954 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:35.546227 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:35.546272 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:35.550415 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:35.550495 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:35.550558 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:35.556161 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:35.557277 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:35.558301 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:35.568167 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:35.568167 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:35.568240 3789 server_base.cc:1034] running on GCE node
W20260927 12:35:35.568168 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:35.568747 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:35.569412 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:35.570592 3789 hybrid_clock.cc:648] HybridClock initialized: now 1790512535570580 us; error 46 us; skew 500 ppm
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:35.573429 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:35.574365 3789 webserver.cc:460] Webserver started at http://127.0.98.3:44377/ using document root <none> and password file <none>
I20260927 12:35:35.574911 3789 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:35.574973 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:35.575256 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:35.577582 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "7cb5d29ba16042b997ebed580be3edd4"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "ca5ddcbd5336772a8ff9f0fe0f260f27"
server_key_iv: "b20c9e2caf269e0924c789dbe7f1b02e"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.578162 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7cb5d29ba16042b997ebed580be3edd4"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "ca5ddcbd5336772a8ff9f0fe0f260f27"
server_key_iv: "b20c9e2caf269e0924c789dbe7f1b02e"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.582322 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:35.585045 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.586393 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:35.586510 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "7cb5d29ba16042b997ebed580be3edd4"
format_stamp: "Formatted at 2026-09-27 12:35:35 on dist-test-slave-tzgx"
server_key: "ca5ddcbd5336772a8ff9f0fe0f260f27"
server_key_iv: "b20c9e2caf269e0924c789dbe7f1b02e"
server_key_version: "encryptionkey@0"
I20260927 12:35:35.586614 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:35.602196 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:35.605263 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:35.605429 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:35.605998 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:35.606806 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:35.606853 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.606894 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:35.606909 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:35.615496 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39477
I20260927 12:35:35.615552 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39477 every 8 connection(s)
I20260927 12:35:35.616458 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:35.622318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3789
I20260927 12:35:35.622442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:35.622700 392 external_mini_cluster.cc:1428] Setting key e077f697791c5d00a5d3dad4250c250d
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:35.629938 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.618161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41279 (local address 127.0.98.62:37625)
0927 12:35:35.618312 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:35.618317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:35.619142 (+ 825us) server_negotiation.cc:407] Connection header received
0927 12:35:35.620471 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:35.620474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:35.620523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:35.620585 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:35.621755 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.622326 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.623654 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.623923 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.624029 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.626571 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:35.626574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:35.626576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:35.626603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:35.627871 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.628418 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.628422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.628423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.628500 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.628845 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.628848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.628849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.629039 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:35.629121 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:35.629657 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 12:35:35.629776 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:35.630803 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.618576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37625 (local address 127.0.98.3:41279)
0927 12:35:35.619001 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:35.619044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:35.620263 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:35.620740 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:35.620747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:35.621110 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:35.621611 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:35.621625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.622482 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.622486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:35.623563 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:35.623573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.624116 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.624122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:35.624282 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.624910 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:35.624932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:35.626388 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:35.627984 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:35.627990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:35.628010 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:35.628326 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:35.628583 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:35.628586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:35.628588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:35.628752 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:35.629134 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:35.629142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:35.629403 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 12:35:35.629587 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:35:35.629934 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260927 12:35:35.632009 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37625
I20260927 12:35:35.632304 3918 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:35.632812 3918 heartbeater.cc:511] Master 127.0.98.62:37625 requested a full tablet report, sending...
I20260927 12:35:35.633811 3459 ts_manager.cc:194] Registered new tserver with Master: 7cb5d29ba16042b997ebed580be3edd4 (127.0.98.3:39477)
I20260927 12:35:35.634374 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41279
I20260927 12:35:35.636622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512535, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:35.663014 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:35.653353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.0.98.62:37625)
0927 12:35:35.653492 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:35.653497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:35.653749 (+ 252us) server_negotiation.cc:407] Connection header received
0927 12:35:35.653846 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:35.653849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:35.653898 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:35.653980 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:35.654770 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.655326 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.656301 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:35.656548 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:35.656633 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:35.659654 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:35.659658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:35.659660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:35.659693 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:35.661143 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.661622 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.661626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.661627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.661720 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:35.662084 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:35.662087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:35.662088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:35.662348 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:35.662474 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:35.662751 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 12:35:35.662860 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 12:35:35.663426 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34390
I20260927 12:35:35.664753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3521
I20260927 12:35:35.670349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3655
I20260927 12:35:35.675074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3789
I20260927 12:35:35.679558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3427
2026-09-27T12:35:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2100 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3930](info): set up 2 sockets
Sep 27 12:35:35 dist-test-slave-tzgx krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260927 12:35:36.643926 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:36 dist-test-slave-tzgx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:36.763700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:36.880343 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:36.880674 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:36.880738 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:36.885159 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:36.885241 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:36.885268 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:36.885289 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:36.885310 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:36.891036 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:36.892254 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:36.893319 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:36.902074 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:36.902074 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:36.902107 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:36.902237 3943 server_base.cc:1034] running on GCE node
I20260927 12:35:36.902926 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:36.904178 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:36.905334 3943 hybrid_clock.cc:648] HybridClock initialized: now 1790512536905297 us; error 54 us; skew 500 ppm
Sep 27 12:35:36 dist-test-slave-tzgx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:36.908330 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:36.909309 3943 webserver.cc:460] Webserver started at http://127.0.98.62:42931/ using document root <none> and password file <none>
I20260927 12:35:36.909891 3943 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:36.909953 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:36.910153 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:36.912563 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "95933d765faf423eae86d1b6571527f0"
format_stamp: "Formatted at 2026-09-27 12:35:36 on dist-test-slave-tzgx"
server_key: "a69cd94cd51a3f77799192f7c8a431a6"
server_key_iv: "72bf3c6fe83d05d5d958aef2813b9b4b"
server_key_version: "encryptionkey@0"
I20260927 12:35:36.913184 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "95933d765faf423eae86d1b6571527f0"
format_stamp: "Formatted at 2026-09-27 12:35:36 on dist-test-slave-tzgx"
server_key: "a69cd94cd51a3f77799192f7c8a431a6"
server_key_iv: "72bf3c6fe83d05d5d958aef2813b9b4b"
server_key_version: "encryptionkey@0"
I20260927 12:35:36.917383 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:36.920511 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:36.922030 3943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:35:36.922187 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "95933d765faf423eae86d1b6571527f0"
format_stamp: "Formatted at 2026-09-27 12:35:36 on dist-test-slave-tzgx"
server_key: "a69cd94cd51a3f77799192f7c8a431a6"
server_key_iv: "72bf3c6fe83d05d5d958aef2813b9b4b"
server_key_version: "encryptionkey@0"
I20260927 12:35:36.922307 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:36.933383 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:36.936579 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:36.936761 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:36.943449 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46853
I20260927 12:35:36.943488 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46853 every 8 connection(s)
I20260927 12:35:36.944437 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:36.947358 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:36.950651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 3943
I20260927 12:35:36.950790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:36.951068 392 external_mini_cluster.cc:1428] Setting key 8cb6f366ff30155d53bbb8dde28e1b8c
I20260927 12:35:36.953974 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0: Bootstrap starting.
I20260927 12:35:36.957059 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:36.958011 4011 log.cc:824] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:36.960253 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0: No bootstrap required, opened a new log
Sep 27 12:35:36 dist-test-slave-tzgx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:36.963865 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95933d765faf423eae86d1b6571527f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } }
I20260927 12:35:36.964205 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:36.964265 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95933d765faf423eae86d1b6571527f0, State: Initialized, Role: FOLLOWER
I20260927 12:35:36.964759 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95933d765faf423eae86d1b6571527f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } }
I20260927 12:35:36.964906 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:36.964980 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:36.965087 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:36.966149 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95933d765faf423eae86d1b6571527f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } }
I20260927 12:35:36.966547 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95933d765faf423eae86d1b6571527f0; no voters:
I20260927 12:35:36.966857 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:36.966964 4016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:36.967166 4016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [term 1 LEADER]: Becoming Leader. State: Replica: 95933d765faf423eae86d1b6571527f0, State: Running, Role: LEADER
I20260927 12:35:36.967446 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:36.952651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.98.62:46853)
0927 12:35:36.953185 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:36.953207 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:36.953256 (+ 49us) server_negotiation.cc:407] Connection header received
0927 12:35:36.954212 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:36.954238 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:36.954712 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:36.955144 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:36.955882 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:36.957444 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:36.958559 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:36.959084 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:36.959241 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:36.961957 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:36.961963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:36.961979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:36.962013 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:36.964297 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:36.964774 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:36.964781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:36.964788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:36.964930 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:36.965225 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:36.965228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:36.965229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:36.965645 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:36.965806 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:36.966134 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 12:35:36.966433 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":137,"threads_started":1}
I20260927 12:35:36.967547 4016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95933d765faf423eae86d1b6571527f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } }
I20260927 12:35:36.968166 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:36.968858 4017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95933d765faf423eae86d1b6571527f0. Latest consensus state: current_term: 1 leader_uuid: "95933d765faf423eae86d1b6571527f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95933d765faf423eae86d1b6571527f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } }
I20260927 12:35:36.968982 4017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:36.970253 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95933d765faf423eae86d1b6571527f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95933d765faf423eae86d1b6571527f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46853 } } }
I20260927 12:35:36.970470 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:36.970608 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:36.973796 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:36.979925 4025 catalog_manager.cc:1347] Generated new cluster ID: 7b3be651462a45b590bddf588b2db814
I20260927 12:35:36.980013 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:37.016639 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:37.017707 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:37.031370 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95933d765faf423eae86d1b6571527f0: Generated new TSK 0
I20260927 12:35:37.032320 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:37.069067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46853
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:37.194154 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:37.194483 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:37.194532 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:37.198871 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:37.198974 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:37.199040 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:37.204792 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:37.205986 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:37.207046 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:37.216856 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:37.216964 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:37.216953 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:37.217016 4037 server_base.cc:1034] running on GCE node
I20260927 12:35:37.217562 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:37.218241 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:37.219414 4037 hybrid_clock.cc:648] HybridClock initialized: now 1790512537219393 us; error 35 us; skew 500 ppm
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:37.222230 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:37.223093 4037 webserver.cc:460] Webserver started at http://127.0.98.1:36593/ using document root <none> and password file <none>
I20260927 12:35:37.223621 4037 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:37.223681 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:37.223845 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:37.226089 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a8b757b0c5f4026942457aa351be2c3"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "9f462d7e115434d12a6621f1e9f6de01"
server_key_iv: "d41797d67b8e0706fd44d6c2195ddd37"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.226657 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a8b757b0c5f4026942457aa351be2c3"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "9f462d7e115434d12a6621f1e9f6de01"
server_key_iv: "d41797d67b8e0706fd44d6c2195ddd37"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.230798 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:35:37.233449 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.234858 4037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:37.234985 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "0a8b757b0c5f4026942457aa351be2c3"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "9f462d7e115434d12a6621f1e9f6de01"
server_key_iv: "d41797d67b8e0706fd44d6c2195ddd37"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.235169 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:37.257105 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:37.261770 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:37.261970 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:37.262603 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:37.263638 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:37.263690 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.263732 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:37.263753 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.272719 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41929
I20260927 12:35:37.272759 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41929 every 8 connection(s)
I20260927 12:35:37.273670 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:37.283162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4037
I20260927 12:35:37.283263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:37.283551 392 external_mini_cluster.cc:1428] Setting key b56c07543b7e1efb004c0bdbc3dcf42b
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:37.288859 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.275523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52077 (local address 127.0.98.62:46853)
0927 12:35:37.275667 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:37.275671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:37.276659 (+ 988us) server_negotiation.cc:407] Connection header received
0927 12:35:37.278074 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:37.278077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:37.278135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:37.278210 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:37.279470 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.280186 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.281849 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.282093 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.282191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.285084 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:37.285088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:37.285091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:37.285120 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:37.286552 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.287100 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.287103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.287104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.287195 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.287631 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.287635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.287636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.287834 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:37.287927 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:37.288544 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 12:35:37.288692 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:35:37.289798 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.275954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46853 (local address 127.0.98.1:52077)
0927 12:35:37.276489 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:37.276535 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:37.277839 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:37.278414 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:37.278422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:37.278799 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:37.279302 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:37.279316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.280324 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.280327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:37.281685 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:37.281698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.282248 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.282252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:37.282430 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.283073 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:37.283090 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:37.284850 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:37.286688 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:37.286695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:37.286712 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:37.287005 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:37.287330 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:37.287332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:37.287334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:37.287507 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:37.287950 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:37.287958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:37.288298 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 12:35:37.288490 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 12:35:37.288866 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":158,"threads_started":1}
I20260927 12:35:37.291164 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46853
I20260927 12:35:37.291435 4166 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:37.292436 4166 heartbeater.cc:511] Master 127.0.98.62:46853 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:35:37.294183 3975 ts_manager.cc:194] Registered new tserver with Master: 0a8b757b0c5f4026942457aa351be2c3 (127.0.98.1:41929)
I20260927 12:35:37.295498 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:37.312057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46853
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:37.431123 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:37.431408 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:37.431450 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:37.435972 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:37.436148 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:37.436240 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:37.442171 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:37.443315 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:37.444399 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:37.454711 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:37.454722 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:37.454916 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:37.455000 4171 server_base.cc:1034] running on GCE node
I20260927 12:35:37.455487 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:37.456387 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:37.457590 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790512537457576 us; error 31 us; skew 500 ppm
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:37.460577 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:37.461501 4171 webserver.cc:460] Webserver started at http://127.0.98.2:41359/ using document root <none> and password file <none>
I20260927 12:35:37.462049 4171 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:37.462111 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:37.462278 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:37.464461 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "64a3fb0cb5bc4afaa02ac5093eb6de17"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "98ace33b7cf45ecefd1d4a42b23ff4ba"
server_key_iv: "88578ab45925b315df46ea4d33945e97"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.465001 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64a3fb0cb5bc4afaa02ac5093eb6de17"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "98ace33b7cf45ecefd1d4a42b23ff4ba"
server_key_iv: "88578ab45925b315df46ea4d33945e97"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.469090 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:37.471846 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.473258 4171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:35:37.473376 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "64a3fb0cb5bc4afaa02ac5093eb6de17"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "98ace33b7cf45ecefd1d4a42b23ff4ba"
server_key_iv: "88578ab45925b315df46ea4d33945e97"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.473510 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:37.485884 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:37.489158 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:37.489336 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:37.489952 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:37.490789 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:37.490844 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.490883 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:37.490897 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.499435 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34939
I20260927 12:35:37.499464 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34939 every 8 connection(s)
I20260927 12:35:37.500422 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:37.509089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4171
I20260927 12:35:37.509220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:37.509516 392 external_mini_cluster.cc:1428] Setting key b286c91156de74e4d73760689815de90
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:37.516506 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.502319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46401 (local address 127.0.98.62:46853)
0927 12:35:37.502460 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:37.502464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:37.503350 (+ 886us) server_negotiation.cc:407] Connection header received
0927 12:35:37.504881 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:37.504884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:37.504947 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:37.505022 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:37.506316 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.506835 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.508071 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.508342 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.508438 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.511470 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:37.511474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:37.511477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:37.511504 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:37.513513 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.514146 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.514150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.514152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.514251 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.514708 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.514712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.514714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.514954 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:37.515072 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:37.516150 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0927 12:35:37.516321 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:35:37.517387 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.502645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46853 (local address 127.0.98.2:46401)
0927 12:35:37.503172 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:37.503223 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:37.504626 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:37.505230 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:37.505239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:37.505647 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:37.506156 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:37.506170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.506975 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.506978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:37.507939 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:37.507951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.508498 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.508502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:37.508678 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.509455 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:37.509473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:37.511170 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:37.513691 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:37.513698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:37.513716 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:37.514032 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:37.514389 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:37.514392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:37.514394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:37.514576 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:37.515176 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:37.515185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:37.515807 (+ 622us) client_negotiation.cc:769] Sending connection context
0927 12:35:37.516030 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 12:35:37.516418 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":123,"threads_started":1}
I20260927 12:35:37.518690 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46853
I20260927 12:35:37.518926 4300 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:37.519469 4300 heartbeater.cc:511] Master 127.0.98.62:46853 requested a full tablet report, sending...
I20260927 12:35:37.520629 3975 ts_manager.cc:194] Registered new tserver with Master: 64a3fb0cb5bc4afaa02ac5093eb6de17 (127.0.98.2:34939)
I20260927 12:35:37.521222 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:37.540342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46853
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:37.650995 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:37.651247 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:37.651305 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:37.655622 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:37.655689 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:37.655769 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:37.661307 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:37.662426 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:37.663460 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:37.674243 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:37.674410 4305 server_base.cc:1034] running on GCE node
W20260927 12:35:37.674242 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:37.674583 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:37.675010 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:37.675724 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:37.676908 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790512537676891 us; error 34 us; skew 500 ppm
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:37.679811 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:37.680771 4305 webserver.cc:460] Webserver started at http://127.0.98.3:40335/ using document root <none> and password file <none>
I20260927 12:35:37.681316 4305 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:37.681380 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:37.681577 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:37.683707 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a99f8f50da748f487a5b55128d7fb91"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "d6d9e7f233edacdcd3154653d5f7bf84"
server_key_iv: "2053a71f7c433a5b8a84daa32233698c"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.684283 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a99f8f50da748f487a5b55128d7fb91"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "d6d9e7f233edacdcd3154653d5f7bf84"
server_key_iv: "2053a71f7c433a5b8a84daa32233698c"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.688292 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:37.690918 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.692245 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:35:37.692373 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "1a99f8f50da748f487a5b55128d7fb91"
format_stamp: "Formatted at 2026-09-27 12:35:37 on dist-test-slave-tzgx"
server_key: "d6d9e7f233edacdcd3154653d5f7bf84"
server_key_iv: "2053a71f7c433a5b8a84daa32233698c"
server_key_version: "encryptionkey@0"
I20260927 12:35:37.692502 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:37.721129 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:37.724251 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:37.724431 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:37.725000 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:37.725873 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:37.725929 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.725983 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:37.726009 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:37.734243 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34765
I20260927 12:35:37.734263 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34765 every 8 connection(s)
I20260927 12:35:37.735140 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:37.737112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4305
I20260927 12:35:37.737231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:37.737498 392 external_mini_cluster.cc:1428] Setting key fcf3cdd819c786f6f93f6c79ffdd95ae
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:37.748986 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.737036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53811 (local address 127.0.98.62:46853)
0927 12:35:37.737256 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:37.737262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:37.738225 (+ 963us) server_negotiation.cc:407] Connection header received
0927 12:35:37.739838 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:37.739841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:37.739898 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:37.739988 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:37.741231 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.741777 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.742981 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.743212 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.743309 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.745793 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:37.745797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:37.745798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:37.745851 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:37.747129 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.747611 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.747614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.747616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.747706 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.748025 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.748027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.748028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.748235 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:37.748308 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:37.748769 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 12:35:37.748865 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:37.749889 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.737531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46853 (local address 127.0.98.3:53811)
0927 12:35:37.738073 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:37.738132 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:37.739559 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:37.740188 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:37.740196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:37.740590 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:37.741088 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:37.741102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.741934 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.741937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:37.742858 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:37.742869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.743372 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.743375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:37.743529 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.744154 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:37.744171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:37.745619 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:37.747236 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:37.747241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:37.747254 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:37.747509 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:37.747795 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:37.747797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:37.747799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:37.747945 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:37.748321 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:37.748327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:37.748578 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 12:35:37.748737 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 12:35:37.749027 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":157,"threads_started":1}
I20260927 12:35:37.751086 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46853
I20260927 12:35:37.751327 4434 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:37.751850 4434 heartbeater.cc:511] Master 127.0.98.62:46853 requested a full tablet report, sending...
I20260927 12:35:37.752905 3972 ts_manager.cc:194] Registered new tserver with Master: 1a99f8f50da748f487a5b55128d7fb91 (127.0.98.3:34765)
I20260927 12:35:37.753525 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53811
I20260927 12:35:37.763923 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:37.789951 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.780334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.98.1:41929)
0927 12:35:37.780646 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:37.780661 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:37.780685 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:35:37.780733 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:37.780738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:37.780924 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:37.781052 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:37.781834 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.782660 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.783542 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.783911 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.783996 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.786488 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:37.786501 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:37.786516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:37.786547 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:37.788178 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.788619 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.788623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.788625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.788703 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.788985 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.788987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.788988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.789260 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:37.789425 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:37.789614 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 12:35:37.789754 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":122,"threads_started":1}
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:37.801915 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:37.793034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.98.62:46853)
0927 12:35:37.793207 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:37.793212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:37.793279 (+ 67us) server_negotiation.cc:407] Connection header received
0927 12:35:37.793434 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:37.793437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:37.793485 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:37.793580 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:37.794248 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.794787 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.795728 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:37.795963 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:37.796049 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:37.798834 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:37.798838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:37.798841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:37.798874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:37.800254 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.800671 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.800674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.800675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.800757 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:37.801067 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:37.801069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:37.801070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:37.801287 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:37.801393 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:37.801639 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 12:35:37.801746 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 12:35:37.802380 3972 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35654
I20260927 12:35:37.803180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4037
I20260927 12:35:37.808204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4171
I20260927 12:35:37.813031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4305
I20260927 12:35:37.817762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 3943
2026-09-27T12:35:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2138 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[4446](info): set up 2 sockets
Sep 27 12:35:37 dist-test-slave-tzgx krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260927 12:35:38.776003 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:38 dist-test-slave-tzgx krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:38.895396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:39.005414 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:39.005710 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:39.005753 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:39.009799 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:39.009868 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:39.009887 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:39.009907 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:39.009922 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:39.015267 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:39.016448 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:39.017447 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:39.025701 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:39.025768 4459 server_base.cc:1034] running on GCE node
W20260927 12:35:39.025696 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.025704 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:39.026507 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:39.027706 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:39.028918 4459 hybrid_clock.cc:648] HybridClock initialized: now 1790512539028887 us; error 47 us; skew 500 ppm
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:39.031973 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:39.032805 4459 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:39.032891 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:39.033098 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:39.035390 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "1212fa9ac6a2436c91d9433ae2d9eab4"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "6cf8d69ba3835213d867f3f16bc5df13"
server_key_iv: "d82641803dabdaf00eb87acfead799fa"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.036010 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "1212fa9ac6a2436c91d9433ae2d9eab4"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "6cf8d69ba3835213d867f3f16bc5df13"
server_key_iv: "d82641803dabdaf00eb87acfead799fa"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.040045 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:35:39.042820 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.044268 4459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:39.044402 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "1212fa9ac6a2436c91d9433ae2d9eab4"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "6cf8d69ba3835213d867f3f16bc5df13"
server_key_iv: "d82641803dabdaf00eb87acfead799fa"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.044520 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:39.054914 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:39.058280 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:39.058459 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:39.065315 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41977
I20260927 12:35:39.065558 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41977 every 8 connection(s)
I20260927 12:35:39.066020 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:39.068913 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:39.071442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4459
I20260927 12:35:39.071559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:39.071821 392 external_mini_cluster.cc:1428] Setting key 46d2fcb189a97839f24dd9db41eff539
I20260927 12:35:39.075317 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4: Bootstrap starting.
I20260927 12:35:39.078163 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:39.079036 4526 log.cc:824] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:39.081449 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4: No bootstrap required, opened a new log
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:39.086095 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260927 12:35:39.086412 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:39.086480 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1212fa9ac6a2436c91d9433ae2d9eab4, State: Initialized, Role: FOLLOWER
I20260927 12:35:39.086992 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260927 12:35:39.087168 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:39.087253 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:39.087347 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:39.087942 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.073381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.0.98.62:41977)
0927 12:35:39.073843 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.073863 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.073908 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:35:39.074702 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.074726 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.075170 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.075561 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:39.076279 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.077754 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.078669 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.079412 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.079560 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.082157 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:39.082164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:39.082182 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:39.082216 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:39.084432 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.084919 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.084926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.084932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.085088 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.085392 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.085396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.085397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.085998 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:39.086213 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.086493 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.086808 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260927 12:35:39.088865 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260927 12:35:39.089257 4526 leader_election.cc:302] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1212fa9ac6a2436c91d9433ae2d9eab4; no voters:
I20260927 12:35:39.089757 4526 leader_election.cc:288] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:39.089869 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:39.090126 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [term 1 LEADER]: Becoming Leader. State: Replica: 1212fa9ac6a2436c91d9433ae2d9eab4, State: Running, Role: LEADER
I20260927 12:35:39.090451 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } }
I20260927 12:35:39.091152 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:39.091568 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1212fa9ac6a2436c91d9433ae2d9eab4. Latest consensus state: current_term: 1 leader_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } } }
I20260927 12:35:39.091683 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:39.092060 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:39.092046 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1212fa9ac6a2436c91d9433ae2d9eab4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41977 } } }
I20260927 12:35:39.093232 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:39.095293 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:39.101253 4535 catalog_manager.cc:1347] Generated new cluster ID: 173f074d145041bfb268b3e84435b131
I20260927 12:35:39.101336 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:39.111999 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:39.113036 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:39.125072 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4: Generated new TSK 0
I20260927 12:35:39.125818 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:39.162513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:39.274055 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:39.274366 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:39.274433 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:39.278677 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:39.278754 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:39.278842 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:39.284459 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:39.285763 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:39.286949 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:39.296928 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.296928 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.297438 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:39.297495 4552 server_base.cc:1034] running on GCE node
I20260927 12:35:39.298151 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:39.298974 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:39.300149 4552 hybrid_clock.cc:648] HybridClock initialized: now 1790512539300072 us; error 93 us; skew 500 ppm
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:39.303406 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:39.304318 4552 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:39.304389 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:39.304579 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:39.306672 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "bcf6007b54324f4197da83c62e9d06a2"
server_key_iv: "eb0d5c5f5a112503e46e638a00b681de"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.307384 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "bcf6007b54324f4197da83c62e9d06a2"
server_key_iv: "eb0d5c5f5a112503e46e638a00b681de"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.312072 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 12:35:39.314579 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.315950 4552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:35:39.316075 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "bcf6007b54324f4197da83c62e9d06a2"
server_key_iv: "eb0d5c5f5a112503e46e638a00b681de"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.316232 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:39.342149 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:39.345643 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:39.345846 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:39.346457 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:39.347309 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:39.347357 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.347404 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:39.347432 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.356508 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40037
I20260927 12:35:39.356530 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40037 every 8 connection(s)
I20260927 12:35:39.357148 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:39.359023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4552
I20260927 12:35:39.359149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:39.359427 392 external_mini_cluster.cc:1428] Setting key 96dc2a517e18656bbdf0a9ec04b72c88
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:35:39.372910 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.358929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38453 (local address 127.0.98.62:41977)
0927 12:35:39.359084 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.359098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.360037 (+ 939us) server_negotiation.cc:407] Connection header received
0927 12:35:39.361512 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.361516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.361569 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.361657 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:39.363331 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.363908 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.365684 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.365950 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.366043 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.368931 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:39.368935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:39.368937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:39.368965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:39.370278 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.370782 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.370785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.370787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.370880 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.371300 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.371307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.371309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.371518 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:39.371596 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.372472 (+ 876us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.372656 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:35:39.373858 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.359362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41977 (local address 127.0.98.1:38453)
0927 12:35:39.359874 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.359929 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.361299 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.361884 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.361891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.362427 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:39.363144 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.363166 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.364042 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.364045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.365524 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.365537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.366121 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.366126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.366335 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.367022 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:39.367043 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:39.368739 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:39.370386 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:39.370393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:39.370409 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:39.370675 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:39.370971 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:39.370975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:39.370977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:39.371176 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:39.371619 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:39.371627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:39.372008 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.372377 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.372766 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":144,"threads_started":1}
I20260927 12:35:39.375275 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41977
I20260927 12:35:39.375530 4680 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:39.376017 4680 heartbeater.cc:511] Master 127.0.98.62:41977 requested a full tablet report, sending...
I20260927 12:35:39.377754 4490 ts_manager.cc:194] Registered new tserver with Master: 8cc873f7c62a42c3ba3b0dc38f5000ba (127.0.98.1:40037)
I20260927 12:35:39.379319 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:39.387689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:39.496061 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:39.496404 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:39.496451 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:39.500614 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:39.500694 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:39.500758 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:39.506615 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:39.507700 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:39.508759 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:39.518469 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.518471 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.518471 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:39.518889 4685 server_base.cc:1034] running on GCE node
I20260927 12:35:39.519554 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:39.520576 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:39.521776 4685 hybrid_clock.cc:648] HybridClock initialized: now 1790512539521751 us; error 42 us; skew 500 ppm
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:39.524890 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:39.525671 4685 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:39.525745 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:39.525949 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:39.528132 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7ba3023a9f24c679ad1e9f8ec80445a"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "d7399d1b67e9cd57a834da564a23eb3a"
server_key_iv: "4cda5c87c4b97d9936600c5170280612"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.528720 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ba3023a9f24c679ad1e9f8ec80445a"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "d7399d1b67e9cd57a834da564a23eb3a"
server_key_iv: "4cda5c87c4b97d9936600c5170280612"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.532611 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:35:39.535115 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.536544 4685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:39.536661 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "b7ba3023a9f24c679ad1e9f8ec80445a"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "d7399d1b67e9cd57a834da564a23eb3a"
server_key_iv: "4cda5c87c4b97d9936600c5170280612"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.536765 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:39.546029 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:39.549036 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:39.549233 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:39.549798 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:39.550601 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:39.550647 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.550698 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:39.550724 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.559275 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36601
I20260927 12:35:39.559291 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36601 every 8 connection(s)
I20260927 12:35:39.559845 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:39.564412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4685
I20260927 12:35:39.564514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:39.564781 392 external_mini_cluster.cc:1428] Setting key fd13b7314dc3e77d821ef07c6009c110
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:39.574028 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.561516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42249 (local address 127.0.98.62:41977)
0927 12:35:39.561641 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.561645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.562439 (+ 794us) server_negotiation.cc:407] Connection header received
0927 12:35:39.563775 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.563778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.563833 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.563905 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:39.565161 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.565768 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.567241 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.567470 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.567557 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.570245 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:39.570251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:39.570254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:39.570285 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:39.571783 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.572350 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.572354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.572356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.572455 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.572771 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.572775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.572777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.573058 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:39.573163 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.573708 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.573848 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:39.574783 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.561888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41977 (local address 127.0.98.2:42249)
0927 12:35:39.562306 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.562347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.563567 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.564116 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.564125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.564498 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:39.565030 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.565045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.565882 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.565884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.567109 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.567121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.567617 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.567622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.567836 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.568442 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:39.568460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:39.570028 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:39.571914 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:39.571920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:39.571934 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:39.572256 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:39.572539 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:39.572542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:39.572543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:39.572677 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:39.573151 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:39.573157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:39.573436 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.573605 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.573881 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260927 12:35:39.575932 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41977
I20260927 12:35:39.576211 4813 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:35:39.576678 4813 heartbeater.cc:511] Master 127.0.98.62:41977 requested a full tablet report, sending...
I20260927 12:35:39.577687 4490 ts_manager.cc:194] Registered new tserver with Master: b7ba3023a9f24c679ad1e9f8ec80445a (127.0.98.2:36601)
I20260927 12:35:39.578423 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:39.592866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:39.706104 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:39.706396 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:39.706442 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:39.710675 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:39.710757 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:39.710827 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:39.716310 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:39.717370 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:39.718390 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:39.728063 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.728063 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:39.728075 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:39.728281 4818 server_base.cc:1034] running on GCE node
I20260927 12:35:39.728760 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:39.729393 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:39.730566 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790512539730546 us; error 35 us; skew 500 ppm
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:39.733826 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:39.734629 4818 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:39.734714 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:39.734926 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:39.737152 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "60ed6918d8ce4805baa1225b8eae734c"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "6ce26cdc684df117c4d08474a5c05ecd"
server_key_iv: "a93af91c969721dfe1af933d0edf6fb4"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.737722 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "60ed6918d8ce4805baa1225b8eae734c"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "6ce26cdc684df117c4d08474a5c05ecd"
server_key_iv: "a93af91c969721dfe1af933d0edf6fb4"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.741927 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:35:39.744374 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.745666 4818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:35:39.745780 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "60ed6918d8ce4805baa1225b8eae734c"
format_stamp: "Formatted at 2026-09-27 12:35:39 on dist-test-slave-tzgx"
server_key: "6ce26cdc684df117c4d08474a5c05ecd"
server_key_iv: "a93af91c969721dfe1af933d0edf6fb4"
server_key_version: "encryptionkey@0"
I20260927 12:35:39.745883 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:39.769790 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:39.773432 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:39.773636 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:39.774250 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:39.775154 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:39.775205 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.775260 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:39.775288 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:39.784138 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42977
I20260927 12:35:39.784168 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42977 every 8 connection(s)
I20260927 12:35:39.784742 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:39.789764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4818
I20260927 12:35:39.789903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:39.790218 392 external_mini_cluster.cc:1428] Setting key 46c846f64267db3deefaae5e8fea74e7
Sep 27 12:35:39 dist-test-slave-tzgx krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:39.799731 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.786528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40259 (local address 127.0.98.62:41977)
0927 12:35:39.786680 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.786685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.787580 (+ 895us) server_negotiation.cc:407] Connection header received
0927 12:35:39.789108 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.789111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.789163 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.789244 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:39.791249 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.792009 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.793494 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.793727 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.793818 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.796308 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:39.796312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:39.796314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:39.796342 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:39.797523 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.798101 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.798104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.798105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.798189 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.798621 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.798624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.798625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.798822 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:39.798946 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.799474 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.799588 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:35:39.800585 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.786917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41977 (local address 127.0.98.3:40259)
0927 12:35:39.787410 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.787457 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.788871 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.789496 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.789508 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.790182 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:39.791115 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.791135 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.792183 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.792187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.793338 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.793350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.793895 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.793899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.794067 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.794665 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:39.794683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:39.796145 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:39.797626 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:39.797631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:39.797645 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:39.797976 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:39.798306 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:39.798309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:39.798310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:39.798491 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:39.798957 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:39.798963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:39.799220 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.799396 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.799666 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260927 12:35:39.801832 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41977
I20260927 12:35:39.802073 4946 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:39.802538 4946 heartbeater.cc:511] Master 127.0.98.62:41977 requested a full tablet report, sending...
I20260927 12:35:39.803691 4490 ts_manager.cc:194] Registered new tserver with Master: 60ed6918d8ce4805baa1225b8eae734c (127.0.98.3:42977)
I20260927 12:35:39.804440 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40259
I20260927 12:35:39.804667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:35:39.813869 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.806632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.0.98.62:41977)
0927 12:35:39.806787 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.806793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.806950 (+ 157us) server_negotiation.cc:407] Connection header received
0927 12:35:39.807057 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.807060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.807106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.807172 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:39.807863 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.808425 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.809261 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.809479 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.809566 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.810853 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:39.810857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:39.810859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:39.810883 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:39.812258 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.812675 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.812679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.812681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.812761 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:39.813040 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:39.813043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:39.813046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:39.813263 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:39.813399 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.813607 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.813715 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:35:39.816738 4490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:39.819774 4490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:39.834522 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.829008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.0.98.2:36601)
0927 12:35:39.829519 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.829531 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.829554 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:39.829708 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.829714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.829899 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.830108 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:39.830290 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.831406 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.833014 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.833798 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.833918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.834079 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.834238 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.834352 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":111,"threads_started":1}
I20260927 12:35:39.834547 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.828204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42977 (local address 127.0.0.1:52778)
0927 12:35:39.828545 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.828600 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.828720 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.829869 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.829875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.829912 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:39.830060 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.830067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.831585 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.831589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.833096 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.833107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.834090 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.834094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.834175 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.834198 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.834248 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.834352 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":175,"spinlock_wait_cycles":8448,"thread_start_us":85,"threads_started":1}
I20260927 12:35:39.834606 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.829132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.98.1:40037)
0927 12:35:39.829428 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.829434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.829839 (+ 405us) server_negotiation.cc:407] Connection header received
0927 12:35:39.830054 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.830059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.830218 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.830333 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:39.830656 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.831401 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.833267 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.833826 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.833926 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.834079 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.834375 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.834473 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":65,"threads_started":1}
I20260927 12:35:39.834888 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.828403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.0.98.3:42977)
0927 12:35:39.829370 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.829382 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.829405 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:39.829480 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.829486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.829704 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.829881 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:39.830183 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.831406 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.833227 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.833963 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.834458 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.834597 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.834647 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.834733 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":181,"threads_started":1}
I20260927 12:35:39.834992 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.828482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36601 (local address 127.0.0.1:60006)
0927 12:35:39.829030 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.829043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.829138 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.830079 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.830083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.830098 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:39.830197 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.830204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.831600 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.831604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.832892 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.832902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.833939 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.833943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.834049 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.834085 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.834200 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.834351 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":110,"thread_start_us":109,"threads_started":1}
I20260927 12:35:39.835243 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.828952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40037 (local address 127.0.0.1:33528)
0927 12:35:39.829756 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.829770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.829916 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.830486 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.830487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.830500 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:39.830558 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.830562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.831517 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.831521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.833139 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.833146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.834135 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.834138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.834227 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.834247 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.834986 (+ 739us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.835133 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":700,"spinlock_wait_cycles":7808,"thread_start_us":89,"threads_started":1}
I20260927 12:35:39.837652 4880 tablet_service.cc:1467] Processing CreateTablet for tablet af2d69bd88ed4c4badf722a21c80567c (DEFAULT_TABLE table=test-table [id=cb185bd0846c496baf64a7d671e511fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:39.838016 4614 tablet_service.cc:1467] Processing CreateTablet for tablet af2d69bd88ed4c4badf722a21c80567c (DEFAULT_TABLE table=test-table [id=cb185bd0846c496baf64a7d671e511fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:39.838018 4747 tablet_service.cc:1467] Processing CreateTablet for tablet af2d69bd88ed4c4badf722a21c80567c (DEFAULT_TABLE table=test-table [id=cb185bd0846c496baf64a7d671e511fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:39.838747 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2d69bd88ed4c4badf722a21c80567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:39.839077 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2d69bd88ed4c4badf722a21c80567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:39.839077 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2d69bd88ed4c4badf722a21c80567c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:39.845391 4963 tablet_bootstrap.cc:492] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Bootstrap starting.
I20260927 12:35:39.846909 4964 tablet_bootstrap.cc:492] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Bootstrap starting.
I20260927 12:35:39.847618 4963 tablet_bootstrap.cc:654] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:39.848439 4965 tablet_bootstrap.cc:492] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: Bootstrap starting.
I20260927 12:35:39.848820 4963 log.cc:824] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:39.848929 4964 tablet_bootstrap.cc:654] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:39.850154 4964 log.cc:824] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:39.850829 4965 tablet_bootstrap.cc:654] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:39.851238 4963 tablet_bootstrap.cc:492] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: No bootstrap required, opened a new log
I20260927 12:35:39.851441 4963 ts_tablet_manager.cc:1397] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:35:39.852113 4965 log.cc:824] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:39.852378 4964 tablet_bootstrap.cc:492] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: No bootstrap required, opened a new log
I20260927 12:35:39.852579 4964 ts_tablet_manager.cc:1397] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:35:39.854293 4965 tablet_bootstrap.cc:492] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: No bootstrap required, opened a new log
I20260927 12:35:39.854496 4965 ts_tablet_manager.cc:1397] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:35:39.855130 4963 raft_consensus.cc:348] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.855449 4963 raft_consensus.cc:374] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:39.855506 4963 raft_consensus.cc:729] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ed6918d8ce4805baa1225b8eae734c, State: Initialized, Role: FOLLOWER
I20260927 12:35:39.855547 4964 raft_consensus.cc:348] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.855803 4964 raft_consensus.cc:374] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:39.855857 4964 raft_consensus.cc:729] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ba3023a9f24c679ad1e9f8ec80445a, State: Initialized, Role: FOLLOWER
I20260927 12:35:39.856007 4963 consensus_queue.cc:260] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.856297 4964 consensus_queue.cc:260] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.856899 4946 heartbeater.cc:503] Master 127.0.98.62:41977 was elected leader, sending a full tablet report...
I20260927 12:35:39.857218 4963 ts_tablet_manager.cc:1428] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:35:39.857908 4964 ts_tablet_manager.cc:1428] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:35:39.858242 4813 heartbeater.cc:503] Master 127.0.98.62:41977 was elected leader, sending a full tablet report...
I20260927 12:35:39.859527 4965 raft_consensus.cc:348] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.859774 4965 raft_consensus.cc:374] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:39.859828 4965 raft_consensus.cc:729] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc873f7c62a42c3ba3b0dc38f5000ba, State: Initialized, Role: FOLLOWER
I20260927 12:35:39.860347 4965 consensus_queue.cc:260] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.861197 4680 heartbeater.cc:503] Master 127.0.98.62:41977 was elected leader, sending a full tablet report...
I20260927 12:35:39.861377 4965 ts_tablet_manager.cc:1428] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 12:35:39.929592 4970 raft_consensus.cc:482] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:39.929847 4970 raft_consensus.cc:504] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.931021 4970 leader_election.cc:288] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ed6918d8ce4805baa1225b8eae734c (127.0.98.3:42977), 8cc873f7c62a42c3ba3b0dc38f5000ba (127.0.98.1:40037)
I20260927 12:35:39.934729 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.931415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43589 (local address 127.0.98.3:42977)
0927 12:35:39.931553 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.931557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.931575 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:35:39.931668 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.931671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.931722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.931834 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:39.932075 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.932600 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.933783 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.934194 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.934289 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.934364 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.934534 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.934610 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:39.934731 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.931280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42977 (local address 127.0.98.2:43589)
0927 12:35:39.931438 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.931454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.931572 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.931838 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.931841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.931859 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:39.931972 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.931979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.932708 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.932712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.933627 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.933637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.934303 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.934306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.934416 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.934438 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.934505 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.934584 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 12:35:39.935006 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.931676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45901 (local address 127.0.98.1:40037)
0927 12:35:39.931808 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.931812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.931944 (+ 132us) server_negotiation.cc:407] Connection header received
0927 12:35:39.932116 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.932119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.932167 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.932230 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:39.932479 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.932988 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.934088 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.934483 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.934581 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.934641 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.934800 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.934881 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:35:39.935175 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.931535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40037 (local address 127.0.98.2:45901)
0927 12:35:39.931867 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.931878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:39.931998 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:39.932247 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:39.932250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:39.932264 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:39.932378 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.932382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.933160 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.933163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.933962 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:39.933971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.934595 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.934598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:39.934692 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.934713 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:35:39.934777 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:35:39.934978 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260927 12:35:39.935449 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2d69bd88ed4c4badf722a21c80567c" candidate_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ed6918d8ce4805baa1225b8eae734c" is_pre_election: true
I20260927 12:35:39.935786 4900 raft_consensus.cc:2454] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7ba3023a9f24c679ad1e9f8ec80445a in term 0.
I20260927 12:35:39.936375 4702 leader_election.cc:302] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ed6918d8ce4805baa1225b8eae734c, b7ba3023a9f24c679ad1e9f8ec80445a; no voters:
I20260927 12:35:39.936658 4970 raft_consensus.cc:2781] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:39.936645 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2d69bd88ed4c4badf722a21c80567c" candidate_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" is_pre_election: true
I20260927 12:35:39.936736 4970 raft_consensus.cc:482] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:39.936769 4970 raft_consensus.cc:3037] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:39.936920 4634 raft_consensus.cc:2454] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7ba3023a9f24c679ad1e9f8ec80445a in term 0.
I20260927 12:35:39.937885 4970 raft_consensus.cc:504] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.938215 4970 leader_election.cc:288] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [CANDIDATE]: Term 1 election: Requested vote from peers 60ed6918d8ce4805baa1225b8eae734c (127.0.98.3:42977), 8cc873f7c62a42c3ba3b0dc38f5000ba (127.0.98.1:40037)
I20260927 12:35:39.938607 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2d69bd88ed4c4badf722a21c80567c" candidate_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ed6918d8ce4805baa1225b8eae734c"
I20260927 12:35:39.938670 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2d69bd88ed4c4badf722a21c80567c" candidate_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba"
I20260927 12:35:39.938778 4634 raft_consensus.cc:3037] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:39.938767 4900 raft_consensus.cc:3037] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:39.940081 4634 raft_consensus.cc:2454] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7ba3023a9f24c679ad1e9f8ec80445a in term 1.
I20260927 12:35:39.940081 4900 raft_consensus.cc:2454] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7ba3023a9f24c679ad1e9f8ec80445a in term 1.
I20260927 12:35:39.940431 4702 leader_election.cc:302] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ed6918d8ce4805baa1225b8eae734c, b7ba3023a9f24c679ad1e9f8ec80445a; no voters:
I20260927 12:35:39.940595 4970 raft_consensus.cc:2781] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:39.940847 4970 raft_consensus.cc:686] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 1 LEADER]: Becoming Leader. State: Replica: b7ba3023a9f24c679ad1e9f8ec80445a, State: Running, Role: LEADER
I20260927 12:35:39.941211 4970 consensus_queue.cc:237] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } }
I20260927 12:35:39.944434 4490 catalog_manager.cc:5495] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a reported cstate change: term changed from 0 to 1, leader changed from <none> to b7ba3023a9f24c679ad1e9f8ec80445a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7ba3023a9f24c679ad1e9f8ec80445a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36601 } health_report { overall_health: HEALTHY } } }
I20260927 12:35:39.957711 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.954037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.0.98.2:36601)
0927 12:35:39.954463 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:39.954466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:39.954478 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:35:39.954551 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:39.954553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:39.954605 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:39.954698 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:39.954931 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.955430 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.956304 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:39.956515 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:39.956601 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:39.956891 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:39.957320 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:39.957426 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:39.957495 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 12:35:39.957551 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260927 12:35:39.963649 4634 raft_consensus.cc:1261] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 1 FOLLOWER]: Refusing update from remote peer b7ba3023a9f24c679ad1e9f8ec80445a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:39.963649 4900 raft_consensus.cc:1261] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 1 FOLLOWER]: Refusing update from remote peer b7ba3023a9f24c679ad1e9f8ec80445a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:39.964385 4973 consensus_queue.cc:1035] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ed6918d8ce4805baa1225b8eae734c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:39.964564 4970 consensus_queue.cc:1035] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc873f7c62a42c3ba3b0dc38f5000ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:39.971395 4976 mvcc.cc:205] Tried to move back new op lower bound from 7333939363683086336 to 7333939363602006016. Current Snapshot: MvccSnapshot[applied={T|T < 7333939363683086336}]
I20260927 12:35:39.972819 4977 mvcc.cc:205] Tried to move back new op lower bound from 7333939363683086336 to 7333939363602006016. Current Snapshot: MvccSnapshot[applied={T|T < 7333939363683086336}]
I20260927 12:35:39.973954 4978 mvcc.cc:205] Tried to move back new op lower bound from 7333939363683086336 to 7333939363602006016. Current Snapshot: MvccSnapshot[applied={T|T < 7333939363683086336}]
I20260927 12:35:39.981334 4490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:39.981534 4490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:39.984159 4490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1212fa9ac6a2436c91d9433ae2d9eab4: Sending DeleteTablet for 3 replicas of tablet af2d69bd88ed4c4badf722a21c80567c
I20260927 12:35:39.985052 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet af2d69bd88ed4c4badf722a21c80567c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33528
I20260927 12:35:39.985051 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet af2d69bd88ed4c4badf722a21c80567c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60006
I20260927 12:35:39.985237 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet af2d69bd88ed4c4badf722a21c80567c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52778
I20260927 12:35:39.985579 4986 tablet_replica.cc:330] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba: stopping tablet replica
I20260927 12:35:39.985579 4987 tablet_replica.cc:330] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: stopping tablet replica
I20260927 12:35:39.985916 4987 raft_consensus.cc:2229] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:39.985937 4986 raft_consensus.cc:2229] T af2d69bd88ed4c4badf722a21c80567c P 8cc873f7c62a42c3ba3b0dc38f5000ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:39.986037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4552
I20260927 12:35:39.986340 4987 raft_consensus.cc:2258] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:39.989070 4987 ts_tablet_manager.cc:1905] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:39.991909 4987 ts_tablet_manager.cc:1918] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:39.992012 4987 log.cc:1196] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/af2d69bd88ed4c4badf722a21c80567c
W20260927 12:35:39.992141 4476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40037 (error 108)
I20260927 12:35:39.992341 4987 ts_tablet_manager.cc:1939] T af2d69bd88ed4c4badf722a21c80567c P b7ba3023a9f24c679ad1e9f8ec80445a: Deleting consensus metadata
I20260927 12:35:39.992360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4685
I20260927 12:35:39.993072 4988 tablet_replica.cc:330] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: stopping tablet replica
I20260927 12:35:39.993405 4988 raft_consensus.cc:2229] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:39.993752 4988 raft_consensus.cc:2258] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:39.994393 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.994059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40037 (local address 127.0.0.1:33542)
0927 12:35:39.994235 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.994307 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 12:35:39.994575 4476 catalog_manager.cc:4584] TS 8cc873f7c62a42c3ba3b0dc38f5000ba (127.0.98.1:40037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af2d69bd88ed4c4badf722a21c80567c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40037: connect: Connection refused (error 111)
I20260927 12:35:39.995184 4988 ts_tablet_manager.cc:1905] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:39.997269 4988 ts_tablet_manager.cc:1918] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:39.997357 4988 log.cc:1196] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/af2d69bd88ed4c4badf722a21c80567c
I20260927 12:35:39.997634 4988 ts_tablet_manager.cc:1939] T af2d69bd88ed4c4badf722a21c80567c P 60ed6918d8ce4805baa1225b8eae734c: Deleting consensus metadata
I20260927 12:35:39.997967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4818
I20260927 12:35:39.999035 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:39.998776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36601 (local address 127.0.0.1:60018)
0927 12:35:39.998892 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:39.998944 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 12:35:40.002992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4459
2026-09-27T12:35:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2184 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T12:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:40Z Disabled control of system clock
I20260927 12:35:40.031144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36231
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 12:35:40.142452 4993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:40.142738 4993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:40.142776 4993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:40.146893 4993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:40.146960 4993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:40.146974 4993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:40.146991 4993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:40.147004 4993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:40.151897 4993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:40.152992 4993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:40.154320 4993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:40.162863 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.162860 4999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:40.162891 4993 server_base.cc:1034] running on GCE node
W20260927 12:35:40.163179 5001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:40.163713 4993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:40.164906 4993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:40.166059 4993 hybrid_clock.cc:648] HybridClock initialized: now 1790512540166037 us; error 43 us; skew 500 ppm
I20260927 12:35:40.167805 4993 webserver.cc:460] Webserver started at http://127.0.98.62:36515/ using document root <none> and password file <none>
I20260927 12:35:40.168423 4993 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:40.168493 4993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:40.168690 4993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:40.170797 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "e55381834f864464a59c54b88e4eb687"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "98e688c6e1325e31b1bcdb6c6a7278ef"
server_key_iv: "71809c2d7deba45cb43ee6462b2f412e"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.171365 4993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "e55381834f864464a59c54b88e4eb687"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "98e688c6e1325e31b1bcdb6c6a7278ef"
server_key_iv: "71809c2d7deba45cb43ee6462b2f412e"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.175221 4993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:40.178041 5007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.179495 4993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:40.179628 4993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "e55381834f864464a59c54b88e4eb687"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "98e688c6e1325e31b1bcdb6c6a7278ef"
server_key_iv: "71809c2d7deba45cb43ee6462b2f412e"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.179750 4993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:40.185105 4993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:40.185820 4993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:40.186002 4993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:40.193101 4993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36231
I20260927 12:35:40.193158 5059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36231 every 8 connection(s)
I20260927 12:35:40.194059 4993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:40.197009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 4993
I20260927 12:35:40.197157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:40.197517 392 external_mini_cluster.cc:1428] Setting key b2cca2eccb18741b9b96f146405852c5
I20260927 12:35:40.197630 5060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:40.203613 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.199077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.0.98.62:36231)
0927 12:35:40.199808 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.199824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.199864 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:35:40.200673 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.200696 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.201191 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.201576 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.202326 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.202332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.202346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.202356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.202412 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.202428 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.202556 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.202632 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.202891 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260927 12:35:40.205173 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687: Bootstrap starting.
I20260927 12:35:40.207541 5060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:40.208346 5060 log.cc:824] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:40.210717 5060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687: No bootstrap required, opened a new log
I20260927 12:35:40.214694 5060 raft_consensus.cc:348] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55381834f864464a59c54b88e4eb687" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260927 12:35:40.214982 5060 raft_consensus.cc:374] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:40.215018 5060 raft_consensus.cc:729] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55381834f864464a59c54b88e4eb687, State: Initialized, Role: FOLLOWER
I20260927 12:35:40.215407 5060 consensus_queue.cc:260] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55381834f864464a59c54b88e4eb687" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260927 12:35:40.215569 5060 raft_consensus.cc:388] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:40.215621 5060 raft_consensus.cc:482] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:40.215698 5060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:40.216759 5060 raft_consensus.cc:504] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55381834f864464a59c54b88e4eb687" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260927 12:35:40.217183 5060 leader_election.cc:302] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e55381834f864464a59c54b88e4eb687; no voters:
I20260927 12:35:40.217562 5060 leader_election.cc:288] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:40.217669 5065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:40.217918 5065 raft_consensus.cc:686] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [term 1 LEADER]: Becoming Leader. State: Replica: e55381834f864464a59c54b88e4eb687, State: Running, Role: LEADER
I20260927 12:35:40.218235 5065 consensus_queue.cc:237] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55381834f864464a59c54b88e4eb687" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } }
I20260927 12:35:40.218828 5060 sys_catalog.cc:564] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:40.219882 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e55381834f864464a59c54b88e4eb687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55381834f864464a59c54b88e4eb687" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } } }
I20260927 12:35:40.220012 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:40.220288 5067 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e55381834f864464a59c54b88e4eb687. Latest consensus state: current_term: 1 leader_uuid: "e55381834f864464a59c54b88e4eb687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55381834f864464a59c54b88e4eb687" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36231 } } }
I20260927 12:35:40.220382 5067 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:40.220503 5074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:40.223750 5074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:40.229894 5074 catalog_manager.cc:1347] Generated new cluster ID: 8f98341514a0479f965d3b7d95334ebf
I20260927 12:35:40.229983 5074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:40.263008 5074 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:40.263381 5074 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:40.272949 5074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687: Generated new TSK 0
I20260927 12:35:40.273778 5074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:35:40.283193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36231
--builtin_ntp_servers=127.0.98.20:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 12:35:40.401398 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:40.401695 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:40.401737 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:40.406960 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:40.407064 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:40.407128 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:40.412921 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:40.414374 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:40.415964 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:40.426223 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.426223 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.426251 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:40.426460 5084 server_base.cc:1034] running on GCE node
I20260927 12:35:40.426978 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:40.427783 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:40.428973 5084 hybrid_clock.cc:648] HybridClock initialized: now 1790512540428946 us; error 39 us; skew 500 ppm
I20260927 12:35:40.434336 5084 webserver.cc:460] Webserver started at http://127.0.98.1:40971/ using document root <none> and password file <none>
I20260927 12:35:40.434978 5084 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:40.435055 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:40.435261 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:40.437588 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "eecd71282b774ac69e8a924510506e42"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "28534d2921b46ed4c8cb7898127da95c"
server_key_iv: "b835e8cbe2de6f26e44920ac5b227528"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.438270 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eecd71282b774ac69e8a924510506e42"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "28534d2921b46ed4c8cb7898127da95c"
server_key_iv: "b835e8cbe2de6f26e44920ac5b227528"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.442597 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:40.445250 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.446586 5084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:40.446702 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "eecd71282b774ac69e8a924510506e42"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "28534d2921b46ed4c8cb7898127da95c"
server_key_iv: "b835e8cbe2de6f26e44920ac5b227528"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.446805 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:40.451818 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:40.452694 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:40.452873 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:40.453666 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:40.454718 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:40.454766 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.454806 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:40.454874 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.463655 5084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42293
I20260927 12:35:40.463680 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42293 every 8 connection(s)
I20260927 12:35:40.464672 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:40.470602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5084
I20260927 12:35:40.470721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:40.470811 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.466564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51463 (local address 127.0.98.62:36231)
0927 12:35:40.466710 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.466714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.467833 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 12:35:40.469155 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.469159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.469204 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.469281 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.470357 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.470382 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.470385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.470398 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.470417 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.470424 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.470508 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.470632 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.470676 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:35:40.471010 392 external_mini_cluster.cc:1428] Setting key 027967030b9e44fee2e152b238578376
I20260927 12:35:40.471740 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.466933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36231 (local address 127.0.98.1:51463)
0927 12:35:40.467647 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:40.467697 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:40.468950 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:40.469454 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:40.469461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:40.469555 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:40.470100 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:40.470102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:40.470144 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:40.470212 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:40.470498 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:40.470506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:40.470512 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:35:40.470667 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 12:35:40.470914 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260927 12:35:40.472906 5212 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36231
I20260927 12:35:40.472909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36231
--builtin_ntp_servers=127.0.98.20:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 12:35:40.474267 5212 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:40.474713 5212 heartbeater.cc:511] Master 127.0.98.62:36231 requested a full tablet report, sending...
I20260927 12:35:40.476334 5024 ts_manager.cc:194] Registered new tserver with Master: eecd71282b774ac69e8a924510506e42 (127.0.98.1:42293)
W20260927 12:35:40.584132 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:40.584430 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:40.584497 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:40.588984 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:40.589062 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:40.589147 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:40.594303 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:40.595626 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:40.597234 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:40.607765 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.607762 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.607765 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:40.608119 5215 server_base.cc:1034] running on GCE node
I20260927 12:35:40.608680 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:40.609426 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:40.610631 5215 hybrid_clock.cc:648] HybridClock initialized: now 1790512540610597 us; error 50 us; skew 500 ppm
I20260927 12:35:40.612762 5215 webserver.cc:460] Webserver started at http://127.0.98.2:43899/ using document root <none> and password file <none>
I20260927 12:35:40.613351 5215 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:40.613420 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:40.613624 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:40.615818 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "25b0b46475424c1e9dbc205785e73690"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "bd49a582e8d7e6e6a9663fb4d1fa1f6f"
server_key_iv: "08b4eb27793a1fc625865e307d498201"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.616402 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "25b0b46475424c1e9dbc205785e73690"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "bd49a582e8d7e6e6a9663fb4d1fa1f6f"
server_key_iv: "08b4eb27793a1fc625865e307d498201"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.620625 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:40.623452 5229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.624939 5215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:40.625101 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "25b0b46475424c1e9dbc205785e73690"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "bd49a582e8d7e6e6a9663fb4d1fa1f6f"
server_key_iv: "08b4eb27793a1fc625865e307d498201"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.625233 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:40.630122 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:40.630786 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:40.630972 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:40.631556 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:40.632718 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:40.632771 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.632812 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:40.632826 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.641479 5215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40411
I20260927 12:35:40.641556 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40411 every 8 connection(s)
I20260927 12:35:40.642475 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:40.648897 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.644487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38573 (local address 127.0.98.62:36231)
0927 12:35:40.644679 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.644683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.645822 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 12:35:40.647173 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.647177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.647224 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.647300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.648450 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.648454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.648457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.648470 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.648492 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.648501 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.648584 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.648703 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.648750 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 12:35:40.649415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5215
I20260927 12:35:40.649533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:40.649777 392 external_mini_cluster.cc:1428] Setting key 97638fa8c2fdcccc834c159efbd03545
I20260927 12:35:40.649883 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.644935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36231 (local address 127.0.98.2:38573)
0927 12:35:40.645654 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:40.645699 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:40.646949 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:40.647494 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:40.647501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:40.647602 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:40.648180 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:40.648182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:40.648224 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:40.648299 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:40.648570 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:40.648578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:40.648584 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:35:40.648747 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 12:35:40.649019 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260927 12:35:40.651162 5343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36231
I20260927 12:35:40.651396 5343 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:40.651619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36231
--builtin_ntp_servers=127.0.98.20:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 12:35:40.651823 5343 heartbeater.cc:511] Master 127.0.98.62:36231 requested a full tablet report, sending...
I20260927 12:35:40.652887 5024 ts_manager.cc:194] Registered new tserver with Master: 25b0b46475424c1e9dbc205785e73690 (127.0.98.2:40411)
W20260927 12:35:40.765138 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:40.765419 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:40.765460 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:40.769842 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:40.769920 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:40.769981 5346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:40.775326 5346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:40.776564 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:40.778013 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:40.788319 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.788304 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:40.788290 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:40.788343 5346 server_base.cc:1034] running on GCE node
I20260927 12:35:40.789009 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:40.789701 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:40.790868 5346 hybrid_clock.cc:648] HybridClock initialized: now 1790512540790846 us; error 35 us; skew 500 ppm
I20260927 12:35:40.792935 5346 webserver.cc:460] Webserver started at http://127.0.98.3:43453/ using document root <none> and password file <none>
I20260927 12:35:40.793532 5346 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:40.793605 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:40.793814 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:40.795964 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "1213fef0cd924c5e9a7cf45bce14bc49"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "fb1b352a2d5f811b03bb99711d655e12"
server_key_iv: "5b5d62f167bebe315e6b08558b74fc9a"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.796581 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1213fef0cd924c5e9a7cf45bce14bc49"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "fb1b352a2d5f811b03bb99711d655e12"
server_key_iv: "5b5d62f167bebe315e6b08558b74fc9a"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.800542 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:40.803288 5360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.804697 5346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:40.804809 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "1213fef0cd924c5e9a7cf45bce14bc49"
format_stamp: "Formatted at 2026-09-27 12:35:40 on dist-test-slave-tzgx"
server_key: "fb1b352a2d5f811b03bb99711d655e12"
server_key_iv: "5b5d62f167bebe315e6b08558b74fc9a"
server_key_version: "encryptionkey@0"
I20260927 12:35:40.804937 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:40.809736 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:40.810416 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:40.810586 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:40.811301 5346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:40.812318 5346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:40.812373 5346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.812413 5346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:40.812427 5346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:40.821146 5346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44177
I20260927 12:35:40.821192 5473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44177 every 8 connection(s)
I20260927 12:35:40.822053 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:40.827929 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.823850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39367 (local address 127.0.98.62:36231)
0927 12:35:40.823988 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.823993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.825132 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 12:35:40.826359 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.826363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.826403 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.826479 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.827528 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.827531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.827534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.827546 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.827567 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.827575 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.827652 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.827747 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.827784 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:35:40.828694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5346
I20260927 12:35:40.828807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:40.829123 392 external_mini_cluster.cc:1428] Setting key d1311f000775ab312991b35b374f7438
I20260927 12:35:40.829114 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.824322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36231 (local address 127.0.98.3:39367)
0927 12:35:40.824983 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:40.825028 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:40.826166 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:40.826644 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:40.826651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:40.826743 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:40.827273 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:40.827274 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:40.827316 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:40.827389 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:40.827642 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:40.827647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:40.827651 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:35:40.827783 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 12:35:40.827994 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260927 12:35:40.830451 5474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36231
I20260927 12:35:40.830716 5474 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:40.831173 5474 heartbeater.cc:511] Master 127.0.98.62:36231 requested a full tablet report, sending...
I20260927 12:35:40.832137 5023 ts_manager.cc:194] Registered new tserver with Master: 1213fef0cd924c5e9a7cf45bce14bc49 (127.0.98.3:44177)
I20260927 12:35:40.843346 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:35:40.846596 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.845229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.0.98.62:36231)
0927 12:35:40.845347 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.845351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.845530 (+ 179us) server_negotiation.cc:407] Connection header received
0927 12:35:40.845631 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.845633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.845668 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.845725 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.846292 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.846294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.846297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.846307 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.846325 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.846332 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.846391 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.846446 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.846479 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:35:40.849357 5023 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:40.852394 5023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:40.864300 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.860837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44177 (local address 127.0.0.1:49568)
0927 12:35:40.861755 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:40.861973 (+ 218us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:40.862135 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:40.862587 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:40.862593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:40.862669 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:40.863244 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:40.863246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:40.863265 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:40.863329 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:40.863909 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:40.863916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:40.863921 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:35:40.864063 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 12:35:40.864157 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":95,"threads_started":1}
I20260927 12:35:40.864673 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.861036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40411 (local address 127.0.0.1:43334)
0927 12:35:40.862221 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:40.862235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:40.862315 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:40.863397 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:40.863401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:40.863417 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:40.863859 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:40.863860 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:40.863869 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:40.863931 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:40.864341 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:40.864345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:40.864348 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:35:40.864512 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:35:40.864547 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":93,"threads_started":1}
I20260927 12:35:40.864711 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.861287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.0.98.3:44177)
0927 12:35:40.862137 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.862153 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.862181 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:35:40.862236 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.862241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.862452 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.862587 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.863446 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.863452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.863463 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.863473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.863509 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.863532 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.863628 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.864478 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.864558 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":165,"threads_started":1}
I20260927 12:35:40.865126 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.861866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.98.2:40411)
0927 12:35:40.862255 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.862266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.862294 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:35:40.862456 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.862462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.862646 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.862775 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.864075 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.864081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.864115 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.864128 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.864171 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.864207 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.864301 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.864877 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.864963 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":96,"threads_started":1}
I20260927 12:35:40.865126 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.861892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.0.98.1:42293)
0927 12:35:40.862488 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:40.862499 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:40.862520 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:35:40.862564 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:40.862568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:40.862693 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:40.862797 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:40.864075 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:40.864081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:40.864115 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:40.864128 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:40.864171 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:40.864207 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:40.864301 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:40.864896 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 12:35:40.864964 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":81,"threads_started":1}
I20260927 12:35:40.864709 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:40.861616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42293 (local address 127.0.0.1:39542)
0927 12:35:40.862347 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:40.862359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:40.862417 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:40.863408 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:40.863411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:40.863421 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:40.863915 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:40.863917 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:40.863924 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:40.863942 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:40.864341 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:40.864345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:40.864348 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:35:40.864597 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 12:35:40.864624 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":94,"threads_started":1}
I20260927 12:35:40.867620 5408 tablet_service.cc:1467] Processing CreateTablet for tablet e6f54c370af54890ae18bd380c75e99d (DEFAULT_TABLE table=test-table [id=7f9db70d82674881829c58c6a0e321d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:40.868043 5277 tablet_service.cc:1467] Processing CreateTablet for tablet e6f54c370af54890ae18bd380c75e99d (DEFAULT_TABLE table=test-table [id=7f9db70d82674881829c58c6a0e321d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:40.868816 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f54c370af54890ae18bd380c75e99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:40.869140 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f54c370af54890ae18bd380c75e99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:40.868911 5146 tablet_service.cc:1467] Processing CreateTablet for tablet e6f54c370af54890ae18bd380c75e99d (DEFAULT_TABLE table=test-table [id=7f9db70d82674881829c58c6a0e321d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:40.869891 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f54c370af54890ae18bd380c75e99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:40.874909 5491 tablet_bootstrap.cc:492] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Bootstrap starting.
I20260927 12:35:40.876324 5492 tablet_bootstrap.cc:492] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Bootstrap starting.
I20260927 12:35:40.877269 5493 tablet_bootstrap.cc:492] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: Bootstrap starting.
I20260927 12:35:40.878182 5491 tablet_bootstrap.cc:654] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:40.878387 5492 tablet_bootstrap.cc:654] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:40.878925 5493 tablet_bootstrap.cc:654] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:40.879068 5492 log.cc:824] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:40.879068 5491 log.cc:824] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:40.879663 5493 log.cc:824] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:40.881147 5491 tablet_bootstrap.cc:492] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: No bootstrap required, opened a new log
I20260927 12:35:40.881146 5492 tablet_bootstrap.cc:492] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: No bootstrap required, opened a new log
I20260927 12:35:40.881270 5493 tablet_bootstrap.cc:492] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: No bootstrap required, opened a new log
I20260927 12:35:40.881345 5492 ts_tablet_manager.cc:1397] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:35:40.881345 5491 ts_tablet_manager.cc:1397] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 12:35:40.881462 5493 ts_tablet_manager.cc:1397] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 12:35:40.885242 5493 raft_consensus.cc:348] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:40.885501 5493 raft_consensus.cc:374] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:40.885555 5493 raft_consensus.cc:729] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecd71282b774ac69e8a924510506e42, State: Initialized, Role: FOLLOWER
I20260927 12:35:40.885789 5491 raft_consensus.cc:348] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:40.885789 5492 raft_consensus.cc:348] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:40.885985 5493 consensus_queue.cc:260] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:40.886054 5491 raft_consensus.cc:374] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:40.886054 5492 raft_consensus.cc:374] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:40.886102 5492 raft_consensus.cc:729] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b0b46475424c1e9dbc205785e73690, State: Initialized, Role: FOLLOWER
I20260927 12:35:40.886102 5491 raft_consensus.cc:729] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1213fef0cd924c5e9a7cf45bce14bc49, State: Initialized, Role: FOLLOWER
I20260927 12:35:40.886530 5492 consensus_queue.cc:260] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:40.886669 5212 heartbeater.cc:503] Master 127.0.98.62:36231 was elected leader, sending a full tablet report...
I20260927 12:35:40.887106 5493 ts_tablet_manager.cc:1428] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:35:40.887169 5343 heartbeater.cc:503] Master 127.0.98.62:36231 was elected leader, sending a full tablet report...
I20260927 12:35:40.887514 5492 ts_tablet_manager.cc:1428] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:35:40.887753 5491 consensus_queue.cc:260] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:40.888386 5474 heartbeater.cc:503] Master 127.0.98.62:36231 was elected leader, sending a full tablet report...
I20260927 12:35:40.889626 5491 ts_tablet_manager.cc:1428] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260927 12:35:40.893571 5344 tablet.cc:2369] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:40.965931 5213 tablet.cc:2369] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:41.073009 5475 tablet.cc:2369] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:41.088493 5498 raft_consensus.cc:482] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:41.088774 5498 raft_consensus.cc:504] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:41.089977 5498 leader_election.cc:288] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1213fef0cd924c5e9a7cf45bce14bc49 (127.0.98.3:44177), eecd71282b774ac69e8a924510506e42 (127.0.98.1:42293)
I20260927 12:35:41.091976 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.090352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42041 (local address 127.0.98.3:44177)
0927 12:35:41.090478 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.090483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.090498 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:35:41.090635 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.090639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.090685 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.090775 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:41.091496 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:41.091500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:41.091503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:41.091515 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:41.091533 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:41.091541 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:41.091630 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:41.091802 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 12:35:41.091846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:41.091980 5345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.090253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44177 (local address 127.0.98.2:42041)
0927 12:35:41.090416 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:41.090432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:41.090521 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:41.090852 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:41.090855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:41.090871 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:41.091332 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:41.091334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:41.091344 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:41.091368 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:41.091626 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:41.091629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:41.091631 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 12:35:41.091792 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 12:35:41.091839 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 12:35:41.092008 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.090285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42293 (local address 127.0.98.2:51719)
0927 12:35:41.090678 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:41.090691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:41.090776 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:41.091006 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:41.091008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:41.091021 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:41.091469 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:35:41.091471 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:41.091480 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:35:41.091504 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:41.091756 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:41.091760 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:41.091764 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:35:41.091837 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:35:41.091860 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":109,"threads_started":1}
I20260927 12:35:41.092012 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.090509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51719 (local address 127.0.98.1:42293)
0927 12:35:41.090629 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.090634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.090770 (+ 136us) server_negotiation.cc:407] Connection header received
0927 12:35:41.090865 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.090868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.090904 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.090982 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:41.091611 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:41.091614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:41.091617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:41.091628 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:41.091648 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:41.091658 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:41.091726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:41.091875 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 12:35:41.091910 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:35:41.092659 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f54c370af54890ae18bd380c75e99d" candidate_uuid: "25b0b46475424c1e9dbc205785e73690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecd71282b774ac69e8a924510506e42" is_pre_election: true
I20260927 12:35:41.092792 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f54c370af54890ae18bd380c75e99d" candidate_uuid: "25b0b46475424c1e9dbc205785e73690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" is_pre_election: true
I20260927 12:35:41.092994 5166 raft_consensus.cc:2454] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25b0b46475424c1e9dbc205785e73690 in term 0.
I20260927 12:35:41.093065 5428 raft_consensus.cc:2454] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25b0b46475424c1e9dbc205785e73690 in term 0.
I20260927 12:35:41.093453 5230 leader_election.cc:302] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1213fef0cd924c5e9a7cf45bce14bc49, 25b0b46475424c1e9dbc205785e73690, eecd71282b774ac69e8a924510506e42; no voters:
I20260927 12:35:41.093729 5498 raft_consensus.cc:2781] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:41.093827 5498 raft_consensus.cc:482] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:41.094084 5498 raft_consensus.cc:3037] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:41.095193 5498 raft_consensus.cc:504] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:41.095501 5498 leader_election.cc:288] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [CANDIDATE]: Term 1 election: Requested vote from peers 1213fef0cd924c5e9a7cf45bce14bc49 (127.0.98.3:44177), eecd71282b774ac69e8a924510506e42 (127.0.98.1:42293)
I20260927 12:35:41.095863 5428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f54c370af54890ae18bd380c75e99d" candidate_uuid: "25b0b46475424c1e9dbc205785e73690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1213fef0cd924c5e9a7cf45bce14bc49"
I20260927 12:35:41.095959 5166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f54c370af54890ae18bd380c75e99d" candidate_uuid: "25b0b46475424c1e9dbc205785e73690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eecd71282b774ac69e8a924510506e42"
I20260927 12:35:41.095979 5428 raft_consensus.cc:3037] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:41.096107 5166 raft_consensus.cc:3037] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:41.097018 5428 raft_consensus.cc:2454] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25b0b46475424c1e9dbc205785e73690 in term 1.
I20260927 12:35:41.097061 5166 raft_consensus.cc:2454] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25b0b46475424c1e9dbc205785e73690 in term 1.
I20260927 12:35:41.097325 5232 leader_election.cc:302] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1213fef0cd924c5e9a7cf45bce14bc49, 25b0b46475424c1e9dbc205785e73690, eecd71282b774ac69e8a924510506e42; no voters:
I20260927 12:35:41.097555 5498 raft_consensus.cc:2781] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:41.097764 5498 raft_consensus.cc:686] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 1 LEADER]: Becoming Leader. State: Replica: 25b0b46475424c1e9dbc205785e73690, State: Running, Role: LEADER
I20260927 12:35:41.098110 5498 consensus_queue.cc:237] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } }
I20260927 12:35:41.101132 5023 catalog_manager.cc:5495] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25b0b46475424c1e9dbc205785e73690 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "25b0b46475424c1e9dbc205785e73690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25b0b46475424c1e9dbc205785e73690" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:41.157756 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.156317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.0.98.2:40411)
0927 12:35:41.156447 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.156450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.156485 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:35:41.156562 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.156564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.156624 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.156685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:41.157423 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:41.157426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:41.157428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:41.157440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:41.157461 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:41.157469 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:41.157552 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:41.157609 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 12:35:41.157643 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:35:41.163683 5428 raft_consensus.cc:1261] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 1 FOLLOWER]: Refusing update from remote peer 25b0b46475424c1e9dbc205785e73690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:41.163683 5166 raft_consensus.cc:1261] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 1 FOLLOWER]: Refusing update from remote peer 25b0b46475424c1e9dbc205785e73690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:41.164472 5501 consensus_queue.cc:1035] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1213fef0cd924c5e9a7cf45bce14bc49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:41.164659 5498 consensus_queue.cc:1035] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eecd71282b774ac69e8a924510506e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:41.172570 5505 mvcc.cc:205] Tried to move back new op lower bound from 7333939368599011328 to 7333939368340148224. Current Snapshot: MvccSnapshot[applied={T|T < 7333939368599011328}]
I20260927 12:35:41.181615 5022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:41.181820 5022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:41.184520 5022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e55381834f864464a59c54b88e4eb687: Sending DeleteTablet for 3 replicas of tablet e6f54c370af54890ae18bd380c75e99d
I20260927 12:35:41.185436 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f54c370af54890ae18bd380c75e99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:41 UTC) from {username='slave'} at 127.0.0.1:49568
I20260927 12:35:41.185441 5277 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f54c370af54890ae18bd380c75e99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:41 UTC) from {username='slave'} at 127.0.0.1:43334
I20260927 12:35:41.185438 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f54c370af54890ae18bd380c75e99d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:41 UTC) from {username='slave'} at 127.0.0.1:39542
I20260927 12:35:41.186018 5514 tablet_replica.cc:330] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: stopping tablet replica
I20260927 12:35:41.186285 5515 tablet_replica.cc:330] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: stopping tablet replica
I20260927 12:35:41.186065 5516 tablet_replica.cc:330] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42: stopping tablet replica
I20260927 12:35:41.186401 5514 raft_consensus.cc:2229] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:41.186764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5084
I20260927 12:35:41.186771 5514 raft_consensus.cc:2258] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:41.187143 5516 raft_consensus.cc:2229] T e6f54c370af54890ae18bd380c75e99d P eecd71282b774ac69e8a924510506e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:41.186765 5515 raft_consensus.cc:2229] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:41.187722 5515 raft_consensus.cc:2258] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:41.188931 5514 ts_tablet_manager.cc:1905] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:41.192263 5514 ts_tablet_manager.cc:1918] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:41.192526 5514 log.cc:1196] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/e6f54c370af54890ae18bd380c75e99d
I20260927 12:35:41.193876 5515 ts_tablet_manager.cc:1905] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:35:41.195806 5008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42293 (error 108)
I20260927 12:35:41.196676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5215
I20260927 12:35:41.197088 5515 ts_tablet_manager.cc:1918] T e6f54c370af54890ae18bd380c75e99d P 25b0b46475424c1e9dbc205785e73690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:41.197655 5514 ts_tablet_manager.cc:1939] T e6f54c370af54890ae18bd380c75e99d P 1213fef0cd924c5e9a7cf45bce14bc49: Deleting consensus metadata
I20260927 12:35:41.199167 5010 catalog_manager.cc:4841] TS 1213fef0cd924c5e9a7cf45bce14bc49 (127.0.98.3:44177): tablet e6f54c370af54890ae18bd380c75e99d (table test-table [id=7f9db70d82674881829c58c6a0e321d3]) successfully deleted
I20260927 12:35:41.199599 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.199269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42293 (local address 127.0.0.1:39558)
0927 12:35:41.199452 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:41.199518 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 12:35:41.199771 5008 catalog_manager.cc:4584] TS eecd71282b774ac69e8a924510506e42 (127.0.98.1:42293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6f54c370af54890ae18bd380c75e99d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42293: connect: Connection refused (error 111)
I20260927 12:35:41.203100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5346
I20260927 12:35:41.203794 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.203457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40411 (local address 127.0.0.1:43352)
0927 12:35:41.203646 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:41.203708 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 12:35:41.208909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 4993
2026-09-27T12:35:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1205 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 12:35:41.229527 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:35:41.231091 392 webserver.cc:460] Webserver started at https://127.0.0.1:46201/ using document root <none> and password file <none>
I20260927 12:35:41.231227 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:35:41.231457 392 webserver.cc:460] Webserver started at http://127.0.0.1:43203/ using document root <none> and password file <none>
2026-09-27T12:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:41Z Disabled control of system clock
I20260927 12:35:41.240044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38285
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38285
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46201/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/testchainca.cert with env {}
W20260927 12:35:41.353121 5523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:41.353425 5523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:41.353482 5523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:41.357717 5523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:41.357797 5523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:41.357820 5523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:41.357851 5523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:41.357872 5523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:35:41.357899 5523 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:35:41.357925 5523 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46201/jwks.json
I20260927 12:35:41.363616 5523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46201/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:41.364784 5523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:41.365823 5523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:41.374387 5529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:41.374538 5523 server_base.cc:1034] running on GCE node
W20260927 12:35:41.374392 5531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:41.374411 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:41.375341 5523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:41.376560 5523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:41.377748 5523 hybrid_clock.cc:648] HybridClock initialized: now 1790512541377726 us; error 35 us; skew 500 ppm
I20260927 12:35:41.379859 5523 webserver.cc:460] Webserver started at http://127.0.98.62:40873/ using document root <none> and password file <none>
I20260927 12:35:41.380499 5523 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:41.380566 5523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:41.380760 5523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:41.382887 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "b5b5eb4fe9d3454f992d7740e127e708"
format_stamp: "Formatted at 2026-09-27 12:35:41 on dist-test-slave-tzgx"
server_key: "3692f86a0851c9898d5462cb884949ab"
server_key_iv: "cc1f5229269a9ca136c246552691db37"
server_key_version: "encryptionkey@0"
I20260927 12:35:41.383417 5523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5b5eb4fe9d3454f992d7740e127e708"
format_stamp: "Formatted at 2026-09-27 12:35:41 on dist-test-slave-tzgx"
server_key: "3692f86a0851c9898d5462cb884949ab"
server_key_iv: "cc1f5229269a9ca136c246552691db37"
server_key_version: "encryptionkey@0"
I20260927 12:35:41.387154 5523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:35:41.389783 5537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:41.391176 5523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:41.391321 5523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "b5b5eb4fe9d3454f992d7740e127e708"
format_stamp: "Formatted at 2026-09-27 12:35:41 on dist-test-slave-tzgx"
server_key: "3692f86a0851c9898d5462cb884949ab"
server_key_iv: "cc1f5229269a9ca136c246552691db37"
server_key_version: "encryptionkey@0"
I20260927 12:35:41.391440 5523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:41.431982 5523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:41.432792 5523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:41.432992 5523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:41.439998 5523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38285
I20260927 12:35:41.440063 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38285 every 8 connection(s)
I20260927 12:35:41.441099 5523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:41.444548 5590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:41.446713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5523
I20260927 12:35:41.446828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:41.447137 392 external_mini_cluster.cc:1428] Setting key 1cb8d240227be3a3a77e48e1a2636381
I20260927 12:35:41.450865 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708: Bootstrap starting.
I20260927 12:35:41.453215 5590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:41.453960 5590 log.cc:824] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:41.456696 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.448677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.0.98.62:38285)
0927 12:35:41.449339 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.449354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.449391 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:35:41.450069 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.450087 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.450513 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.450889 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:41.451343 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.452386 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.453275 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.453814 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.453964 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:41.454830 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:41.454836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:41.454849 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:41.455248 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:41.455304 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:41.455325 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:41.455410 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:41.455634 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 12:35:41.455911 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260927 12:35:41.456784 5590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708: No bootstrap required, opened a new log
I20260927 12:35:41.459450 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:35:41.460678 5590 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b5eb4fe9d3454f992d7740e127e708" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38285 } }
I20260927 12:35:41.460947 5590 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:41.461009 5590 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b5eb4fe9d3454f992d7740e127e708, State: Initialized, Role: FOLLOWER
I20260927 12:35:41.461417 5590 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b5eb4fe9d3454f992d7740e127e708" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38285 } }
I20260927 12:35:41.461551 5590 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:41.461611 5590 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:41.461706 5590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:41.462785 5590 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b5eb4fe9d3454f992d7740e127e708" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38285 } }
I20260927 12:35:41.463147 5590 leader_election.cc:302] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b5eb4fe9d3454f992d7740e127e708; no voters:
I20260927 12:35:41.463464 5590 leader_election.cc:288] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:41.463573 5595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:41.463791 5595 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [term 1 LEADER]: Becoming Leader. State: Replica: b5b5eb4fe9d3454f992d7740e127e708, State: Running, Role: LEADER
I20260927 12:35:41.464150 5595 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b5eb4fe9d3454f992d7740e127e708" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38285 } }
I20260927 12:35:41.464749 5590 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:41.465736 5597 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b5eb4fe9d3454f992d7740e127e708. Latest consensus state: current_term: 1 leader_uuid: "b5b5eb4fe9d3454f992d7740e127e708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b5eb4fe9d3454f992d7740e127e708" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38285 } } }
I20260927 12:35:41.465868 5597 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:41.466117 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5b5eb4fe9d3454f992d7740e127e708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b5eb4fe9d3454f992d7740e127e708" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38285 } } }
I20260927 12:35:41.466202 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:41.466301 5604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:41.469496 5604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:41.475471 5604 catalog_manager.cc:1347] Generated new cluster ID: 37be95d7f13e492485a8a154d7b80fca
I20260927 12:35:41.475571 5604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:41.497125 5604 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:41.498389 5604 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:41.508214 5604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5b5eb4fe9d3454f992d7740e127e708: Generated new TSK 0
I20260927 12:35:41.509172 5604 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 12:35:41.540701 5620 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 12:35:41.540832 5620 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 12:35:41.541421 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.524125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.0.98.62:38285)
0927 12:35:41.524255 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.524259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.524475 (+ 216us) server_negotiation.cc:407] Connection header received
0927 12:35:41.524708 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.524711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.524761 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.524847 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:35:41.525116 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.525702 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.526594 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.526822 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.526911 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:41.527347 (+ 436us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:35:41.540995 (+ 13648us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46201/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 12:35:41.541280 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46201/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 12:35:41.541529 5593 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46201/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 12:35:41.542622 5619 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:41.523940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38285 (local address 127.0.0.1:41886)
0927 12:35:41.524386 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:41.524398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:41.524533 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:41.524889 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:41.524892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:41.524901 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:35:41.525012 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:41.525018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.525837 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.525840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:41.526478 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:41.526486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.526915 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.526942 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:41.527021 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:41.527073 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:35:41.541229 (+ 14156us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46201/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 12:35:41.541466 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46201/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46201/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 12:35:41.543470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5523
2026-09-27T12:35:41Z chronyd exiting
I20260927 12:35:41.580701 392 test_util.cc:175] -----------------------------------------------
I20260927 12:35:41.580797 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512512765972-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (352 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 12:35:41.581943 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:35:41.582872 392 webserver.cc:460] Webserver started at https://127.0.0.1:44389/ using document root <none> and password file <none>
I20260927 12:35:41.582985 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:35:41.583262 392 webserver.cc:460] Webserver started at http://127.0.0.1:43501/ using document root <none> and password file <none>
2026-09-27T12:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:41Z Disabled control of system clock
I20260927 12:35:41.592263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38239
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38239
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44389/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/testchainca.cert with env {}
W20260927 12:35:41.708557 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:41.708835 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:41.708877 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:41.712998 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:41.713069 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:41.713083 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:41.713101 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:41.713115 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:35:41.713130 5626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:35:41.713142 5626 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44389/jwks.json
I20260927 12:35:41.718762 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44389/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:41.719830 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:41.720865 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:41.729398 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:41.729408 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:41.729420 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:41.729791 5626 server_base.cc:1034] running on GCE node
I20260927 12:35:41.730391 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:41.731504 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:41.732692 5626 hybrid_clock.cc:648] HybridClock initialized: now 1790512541732674 us; error 31 us; skew 500 ppm
I20260927 12:35:41.734772 5626 webserver.cc:460] Webserver started at http://127.0.98.62:34055/ using document root <none> and password file <none>
I20260927 12:35:41.735369 5626 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:41.735440 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:41.735678 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:41.737915 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "27a8e42f39b34b86a53e122ba9c1d33e"
format_stamp: "Formatted at 2026-09-27 12:35:41 on dist-test-slave-tzgx"
server_key: "6e05012e4aa626201866a9a079e49ce3"
server_key_iv: "e84694cc9ce536685dbb34148df9b83d"
server_key_version: "encryptionkey@0"
I20260927 12:35:41.738583 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "27a8e42f39b34b86a53e122ba9c1d33e"
format_stamp: "Formatted at 2026-09-27 12:35:41 on dist-test-slave-tzgx"
server_key: "6e05012e4aa626201866a9a079e49ce3"
server_key_iv: "e84694cc9ce536685dbb34148df9b83d"
server_key_version: "encryptionkey@0"
I20260927 12:35:41.743085 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 12:35:41.745958 5640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:41.747370 5626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:41.747500 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "27a8e42f39b34b86a53e122ba9c1d33e"
format_stamp: "Formatted at 2026-09-27 12:35:41 on dist-test-slave-tzgx"
server_key: "6e05012e4aa626201866a9a079e49ce3"
server_key_iv: "e84694cc9ce536685dbb34148df9b83d"
server_key_version: "encryptionkey@0"
I20260927 12:35:41.747635 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:41.794335 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:41.795030 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:41.795186 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:41.802423 5626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38239
I20260927 12:35:41.802449 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38239 every 8 connection(s)
I20260927 12:35:41.803390 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:41.806456 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:41.808934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5626
I20260927 12:35:41.809047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:41.809309 392 external_mini_cluster.cc:1428] Setting key 442f2b04608c0c0a324c838a53ceb6c9
I20260927 12:35:41.812861 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e: Bootstrap starting.
I20260927 12:35:41.815608 5693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:41.816412 5693 log.cc:824] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:41.818663 5693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e: No bootstrap required, opened a new log
I20260927 12:35:41.819867 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.810837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.0.98.62:38239)
0927 12:35:41.811556 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.811575 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.811620 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:35:41.812466 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.812490 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.813019 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.813393 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:41.813847 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.815314 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.816351 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.816988 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.817117 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:41.817992 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:41.817998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:41.818023 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:41.818502 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:41.818576 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:41.818602 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:41.818698 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:41.818920 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 12:35:41.819176 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
I20260927 12:35:41.822743 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:35:41.822758 5693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260927 12:35:41.823015 5693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:41.823064 5693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a8e42f39b34b86a53e122ba9c1d33e, State: Initialized, Role: FOLLOWER
I20260927 12:35:41.823493 5693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260927 12:35:41.823661 5693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:41.823725 5693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:41.823823 5693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:41.824851 5693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260927 12:35:41.825167 5693 leader_election.cc:302] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27a8e42f39b34b86a53e122ba9c1d33e; no voters:
I20260927 12:35:41.825510 5693 leader_election.cc:288] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:41.825577 5698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:41.825793 5698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [term 1 LEADER]: Becoming Leader. State: Replica: 27a8e42f39b34b86a53e122ba9c1d33e, State: Running, Role: LEADER
I20260927 12:35:41.826124 5698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } }
I20260927 12:35:41.826653 5693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:41.827644 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27a8e42f39b34b86a53e122ba9c1d33e. Latest consensus state: current_term: 1 leader_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } } }
I20260927 12:35:41.827762 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:41.828137 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27a8e42f39b34b86a53e122ba9c1d33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38239 } } }
I20260927 12:35:41.828287 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:41.828877 5707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:41.831363 5707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:41.836793 5707 catalog_manager.cc:1347] Generated new cluster ID: 8079568bddf84f02a6b24fa60828e848
I20260927 12:35:41.836863 5707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:41.847479 5707 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:41.848454 5707 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:41.856962 5707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27a8e42f39b34b86a53e122ba9c1d33e: Generated new TSK 0
I20260927 12:35:41.857697 5707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 12:35:41.890774 5723 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 12:35:41.890849 5723 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 12:35:41.891620 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:41.875609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.0.98.62:38239)
0927 12:35:41.875746 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:41.875751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:41.875865 (+ 114us) server_negotiation.cc:407] Connection header received
0927 12:35:41.876030 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:41.876034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:41.876130 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:41.876257 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:35:41.876452 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.877281 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.878172 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.878530 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.878629 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:41.878877 (+ 248us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:35:41.891183 (+ 12306us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 12:35:41.891476 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 12:35:41.891692 5696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 12:35:41.891717 5722 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:41.875462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38239 (local address 127.0.0.1:58904)
0927 12:35:41.875774 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:41.875787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:41.875889 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:41.876247 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:41.876249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:41.876255 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:35:41.876353 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:41.876358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.877397 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.877400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:41.878066 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:41.878074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:41.878611 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:41.878632 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:41.878695 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:41.878718 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:35:41.891392 (+ 12674us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 12:35:41.891567 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38239: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44389/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
I20260927 12:35:41.892545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5626
2026-09-27T12:35:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 12:35:41.930835 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:35:41.931677 392 webserver.cc:460] Webserver started at https://127.0.0.1:38399/ using document root <none> and password file <none>
I20260927 12:35:41.931761 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:35:41.931981 392 webserver.cc:460] Webserver started at http://127.0.0.1:43719/ using document root <none> and password file <none>
2026-09-27T12:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:41Z Disabled control of system clock
I20260927 12:35:41.939854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44349
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 12:35:42.049150 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:42.049444 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:42.049497 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:42.053716 5729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:42.053802 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:42.053830 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:42.053851 5729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:42.053872 5729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:35:42.053902 5729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:35:42.053926 5729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38399/jwks.json
I20260927 12:35:42.058920 5729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:42.060082 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:42.061542 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:42.069857 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:42.069869 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:42.069869 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:42.070120 5729 server_base.cc:1034] running on GCE node
I20260927 12:35:42.070863 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:42.072120 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:42.073320 5729 hybrid_clock.cc:648] HybridClock initialized: now 1790512542073302 us; error 33 us; skew 500 ppm
I20260927 12:35:42.075402 5729 webserver.cc:460] Webserver started at http://127.0.98.62:44577/ using document root <none> and password file <none>
I20260927 12:35:42.076036 5729 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:42.076207 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:42.076455 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:42.078537 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "20ca24c1067544d3851df58d5905b068"
format_stamp: "Formatted at 2026-09-27 12:35:42 on dist-test-slave-tzgx"
server_key: "99aa17674215fe491f45d99d23336f97"
server_key_iv: "34df1ca4b402c6af65cd55aaa20265e3"
server_key_version: "encryptionkey@0"
I20260927 12:35:42.079094 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "20ca24c1067544d3851df58d5905b068"
format_stamp: "Formatted at 2026-09-27 12:35:42 on dist-test-slave-tzgx"
server_key: "99aa17674215fe491f45d99d23336f97"
server_key_iv: "34df1ca4b402c6af65cd55aaa20265e3"
server_key_version: "encryptionkey@0"
I20260927 12:35:42.082990 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 12:35:42.085879 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:42.087323 5729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:42.087471 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "20ca24c1067544d3851df58d5905b068"
format_stamp: "Formatted at 2026-09-27 12:35:42 on dist-test-slave-tzgx"
server_key: "99aa17674215fe491f45d99d23336f97"
server_key_iv: "34df1ca4b402c6af65cd55aaa20265e3"
server_key_version: "encryptionkey@0"
I20260927 12:35:42.087598 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:42.115062 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:42.115914 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:42.116420 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:42.124670 5729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44349
I20260927 12:35:42.124696 5795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44349 every 8 connection(s)
I20260927 12:35:42.125679 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:42.129094 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:42.135627 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068: Bootstrap starting.
I20260927 12:35:42.135927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5729
I20260927 12:35:42.136040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:42.136328 392 external_mini_cluster.cc:1428] Setting key b3803d4d683fd463356ff3b7091945bd
I20260927 12:35:42.138574 5796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:42.139336 5796 log.cc:824] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:42.141484 5796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068: No bootstrap required, opened a new log
I20260927 12:35:42.145951 5796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ca24c1067544d3851df58d5905b068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44349 } }
I20260927 12:35:42.146202 5796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:42.146236 5796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ca24c1067544d3851df58d5905b068, State: Initialized, Role: FOLLOWER
I20260927 12:35:42.146687 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:42.137793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.0.98.62:44349)
0927 12:35:42.138506 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:42.138524 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:42.138562 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:35:42.139303 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:42.139324 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:42.139831 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:42.140265 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:42.140637 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.141632 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.142898 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.143449 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.143588 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:42.144654 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:42.144660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:42.144677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:42.145123 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:42.145186 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:42.145207 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:42.145290 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:42.145497 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:35:42.145809 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":142,"threads_started":1}
I20260927 12:35:42.146670 5796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ca24c1067544d3851df58d5905b068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44349 } }
I20260927 12:35:42.146781 5796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:42.146833 5796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:42.146926 5796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:42.148128 5796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ca24c1067544d3851df58d5905b068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44349 } }
I20260927 12:35:42.148475 5796 leader_election.cc:302] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ca24c1067544d3851df58d5905b068; no voters:
I20260927 12:35:42.148957 5796 leader_election.cc:288] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:42.149051 5801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:42.149262 5801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [term 1 LEADER]: Becoming Leader. State: Replica: 20ca24c1067544d3851df58d5905b068, State: Running, Role: LEADER
I20260927 12:35:42.149605 5801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ca24c1067544d3851df58d5905b068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44349 } }
I20260927 12:35:42.150024 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:35:42.150307 5796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:42.151314 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ca24c1067544d3851df58d5905b068. Latest consensus state: current_term: 1 leader_uuid: "20ca24c1067544d3851df58d5905b068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ca24c1067544d3851df58d5905b068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44349 } } }
I20260927 12:35:42.151440 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:42.151731 5802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20ca24c1067544d3851df58d5905b068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ca24c1067544d3851df58d5905b068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44349 } } }
I20260927 12:35:42.151821 5802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:42.152187 5810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:42.154865 5810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:42.160477 5810 catalog_manager.cc:1347] Generated new cluster ID: 3b810a8cfc33492aafed0d44e4552640
I20260927 12:35:42.160561 5810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:42.169641 5810 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:42.170583 5810 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:42.176494 5810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20ca24c1067544d3851df58d5905b068: Generated new TSK 0
I20260927 12:35:42.177228 5810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:35:42.190704 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:42.185077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.0.98.62:44349)
0927 12:35:42.185214 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:42.185219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:42.185339 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:35:42.185498 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:42.185501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:42.185566 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:42.185698 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:35:42.186019 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.186840 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.187856 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.188239 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.188374 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:42.188676 (+ 302us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:35:42.190230 (+ 1554us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 12:35:42.190528 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44542: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 12:35:42.190779 5799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44542: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 12:35:42.192515 5825 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:42.184974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44349 (local address 127.0.0.1:44542)
0927 12:35:42.185252 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:42.185265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:42.185362 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:42.185769 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:42.185772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:42.185781 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:35:42.185896 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:42.185902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.186990 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.186997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:42.187685 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:42.187697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.188359 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.188381 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:42.188465 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:42.188493 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:35:42.190403 (+ 1910us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 12:35:42.192326 (+ 1923us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1592,"thread_start_us":91,"threads_started":1}
I20260927 12:35:42.193114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5729
E20260927 12:35:42.201853 5826 webserver.cc:533] Webserver: sslize failed:
E20260927 12:35:42.201972 5826 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T12:35:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T12:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:42Z Disabled control of system clock
I20260927 12:35:42.239853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35267
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35267
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/testchainca.cert with env {}
W20260927 12:35:42.351946 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:42.352329 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:42.352375 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:42.356626 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:42.356700 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:42.356714 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:42.356730 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:42.356745 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:35:42.356760 5830 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:35:42.356775 5830 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 12:35:42.362743 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:42.363879 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:42.364939 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:42.373485 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:42.373562 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:42.373617 5830 server_base.cc:1034] running on GCE node
W20260927 12:35:42.373572 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:42.374305 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:42.375435 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:42.376596 5830 hybrid_clock.cc:648] HybridClock initialized: now 1790512542376573 us; error 41 us; skew 500 ppm
I20260927 12:35:42.378582 5830 webserver.cc:460] Webserver started at http://127.0.98.62:40313/ using document root <none> and password file <none>
I20260927 12:35:42.379172 5830 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:42.379241 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:42.379441 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:42.381711 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "affafa6bd13e45ad94076cfffdeef8f4"
format_stamp: "Formatted at 2026-09-27 12:35:42 on dist-test-slave-tzgx"
server_key: "78bbc304d5f6a587aaf4622caad92366"
server_key_iv: "2616dd7147206b13e5c5859a469849c5"
server_key_version: "encryptionkey@0"
I20260927 12:35:42.382314 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "affafa6bd13e45ad94076cfffdeef8f4"
format_stamp: "Formatted at 2026-09-27 12:35:42 on dist-test-slave-tzgx"
server_key: "78bbc304d5f6a587aaf4622caad92366"
server_key_iv: "2616dd7147206b13e5c5859a469849c5"
server_key_version: "encryptionkey@0"
I20260927 12:35:42.386394 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:42.389086 5844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:42.390524 5830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:42.390662 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "affafa6bd13e45ad94076cfffdeef8f4"
format_stamp: "Formatted at 2026-09-27 12:35:42 on dist-test-slave-tzgx"
server_key: "78bbc304d5f6a587aaf4622caad92366"
server_key_iv: "2616dd7147206b13e5c5859a469849c5"
server_key_version: "encryptionkey@0"
I20260927 12:35:42.390798 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:42.423288 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:42.424014 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:42.424216 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:42.430999 5830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35267
I20260927 12:35:42.431058 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35267 every 8 connection(s)
I20260927 12:35:42.431950 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:42.435204 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:42.436301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5830
I20260927 12:35:42.436419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:42.436721 392 external_mini_cluster.cc:1428] Setting key 5291e92effdc8fad80de480680f3094c
I20260927 12:35:42.441548 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4: Bootstrap starting.
I20260927 12:35:42.443998 5897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:42.444779 5897 log.cc:824] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:42.446240 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:42.438130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.0.98.62:35267)
0927 12:35:42.438819 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:42.438837 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:42.438874 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:35:42.439524 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:42.439541 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:42.439972 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:42.440333 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:42.440785 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.441791 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.442693 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.443183 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.443343 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:42.444266 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:42.444271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:42.444285 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:35:42.444692 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:42.444753 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:35:42.444774 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:42.444858 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:42.445048 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 12:35:42.445333 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260927 12:35:42.446928 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4: No bootstrap required, opened a new log
I20260927 12:35:42.449213 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:35:42.451217 5897 raft_consensus.cc:348] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affafa6bd13e45ad94076cfffdeef8f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35267 } }
I20260927 12:35:42.451464 5897 raft_consensus.cc:374] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:42.451514 5897 raft_consensus.cc:729] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affafa6bd13e45ad94076cfffdeef8f4, State: Initialized, Role: FOLLOWER
I20260927 12:35:42.451897 5897 consensus_queue.cc:260] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affafa6bd13e45ad94076cfffdeef8f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35267 } }
I20260927 12:35:42.452013 5897 raft_consensus.cc:388] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:42.452052 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:42.452152 5897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:42.453081 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affafa6bd13e45ad94076cfffdeef8f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35267 } }
I20260927 12:35:42.453408 5897 leader_election.cc:302] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affafa6bd13e45ad94076cfffdeef8f4; no voters:
I20260927 12:35:42.453688 5897 leader_election.cc:288] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:42.453886 5902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:42.454109 5902 raft_consensus.cc:686] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [term 1 LEADER]: Becoming Leader. State: Replica: affafa6bd13e45ad94076cfffdeef8f4, State: Running, Role: LEADER
I20260927 12:35:42.454784 5902 consensus_queue.cc:237] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affafa6bd13e45ad94076cfffdeef8f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35267 } }
I20260927 12:35:42.455055 5897 sys_catalog.cc:564] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:42.456593 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader affafa6bd13e45ad94076cfffdeef8f4. Latest consensus state: current_term: 1 leader_uuid: "affafa6bd13e45ad94076cfffdeef8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affafa6bd13e45ad94076cfffdeef8f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35267 } } }
I20260927 12:35:42.456632 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "affafa6bd13e45ad94076cfffdeef8f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affafa6bd13e45ad94076cfffdeef8f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35267 } } }
I20260927 12:35:42.456733 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:42.456759 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4 [sys.catalog]: This master's current role is: LEADER
W20260927 12:35:42.459858 5917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:42.459935 5917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:42.460012 5916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:42.460723 5916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:42.466702 5916 catalog_manager.cc:1347] Generated new cluster ID: adb2d9bcd1e94e439482bf545b3f37ee
I20260927 12:35:42.466773 5916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:42.483033 5916 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:42.483981 5916 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:42.488484 5916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P affafa6bd13e45ad94076cfffdeef8f4: Generated new TSK 0
I20260927 12:35:42.489249 5916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:35:42.560766 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:42.498303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.0.98.62:35267)
0927 12:35:42.498426 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:42.498430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:42.498586 (+ 156us) server_negotiation.cc:407] Connection header received
0927 12:35:42.498756 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:42.498759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:42.498825 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:42.498933 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:35:42.499221 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.499787 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.501042 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.501267 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.501372 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:42.501700 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:35:42.560376 (+ 58676us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 12:35:42.560643 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39290: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 12:35:42.560843 5900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39290: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 12:35:42.561057 5926 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:42.498146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35267 (local address 127.0.0.1:39290)
0927 12:35:42.498477 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:42.498492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:42.498617 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:42.498941 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:42.498943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:42.498951 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:35:42.499080 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:42.499087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.499892 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.499896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:42.500899 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:42.500910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:42.501373 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:42.501397 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:42.501492 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:42.501521 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:35:42.560630 (+ 59109us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 12:35:42.560860 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":127,"threads_started":1}
I20260927 12:35:42.561683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5830
2026-09-27T12:35:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (348 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:42 dist-test-slave-tzgx krb5kdc[5938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:42 dist-test-slave-tzgx krb5kdc[5938](info): set up 2 sockets
Sep 27 12:35:42 dist-test-slave-tzgx krb5kdc[5938](info): commencing operation
krb5kdc: starting...
W20260927 12:35:43.788280 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:43 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:43.910883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:44.026566 5951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:44.026862 5951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:44.026903 5951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:44.031119 5951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:44.031198 5951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:44.031216 5951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:44.031232 5951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:44.031246 5951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:44.036935 5951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:44.038173 5951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:44.039289 5951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:44.047920 5956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:44.047960 5957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:44.047960 5951 server_base.cc:1034] running on GCE node
W20260927 12:35:44.048163 5959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:44.048679 5951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:44.049801 5951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:44.050963 5951 hybrid_clock.cc:648] HybridClock initialized: now 1790512544050953 us; error 42 us; skew 500 ppm
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.053736 5951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:44.054697 5951 webserver.cc:460] Webserver started at http://127.0.98.62:40607/ using document root <none> and password file <none>
I20260927 12:35:44.055238 5951 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:44.055299 5951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:44.055480 5951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:44.057698 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "c4d14a038781486ab4968fd7c81037a3"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "6a31e9fc62e51d8a1db70cafbdf90f63"
server_key_iv: "df4cd814c23e9ceb62cbd540e8b215ee"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.058252 5951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4d14a038781486ab4968fd7c81037a3"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "6a31e9fc62e51d8a1db70cafbdf90f63"
server_key_iv: "df4cd814c23e9ceb62cbd540e8b215ee"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.061977 5951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:35:44.064513 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.065901 5951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:44.066051 5951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "c4d14a038781486ab4968fd7c81037a3"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "6a31e9fc62e51d8a1db70cafbdf90f63"
server_key_iv: "df4cd814c23e9ceb62cbd540e8b215ee"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.066161 5951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:44.078168 5951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:44.081032 5951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:44.081202 5951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:44.087913 5951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43489
I20260927 12:35:44.088234 6018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43489 every 8 connection(s)
I20260927 12:35:44.088958 5951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:44.091812 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:44.097461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 5951
I20260927 12:35:44.097577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:44.097754 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3: Bootstrap starting.
I20260927 12:35:44.097812 392 external_mini_cluster.cc:1428] Setting key 401bc3d648cf37a0379d268597d32549
I20260927 12:35:44.100451 6019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:44.101276 6019 log.cc:824] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:44.103472 6019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3: No bootstrap required, opened a new log
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:44.107890 6019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d14a038781486ab4968fd7c81037a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43489 } }
I20260927 12:35:44.108199 6019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:44.108269 6019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d14a038781486ab4968fd7c81037a3, State: Initialized, Role: FOLLOWER
I20260927 12:35:44.108809 6019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d14a038781486ab4968fd7c81037a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43489 } }
I20260927 12:35:44.108994 6019 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:44.109068 6019 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:44.109177 6019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:44.110342 6019 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d14a038781486ab4968fd7c81037a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43489 } }
I20260927 12:35:44.110721 6019 leader_election.cc:302] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d14a038781486ab4968fd7c81037a3; no voters:
I20260927 12:35:44.111117 6019 leader_election.cc:288] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:44.111227 6024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:44.111439 6024 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [term 1 LEADER]: Becoming Leader. State: Replica: c4d14a038781486ab4968fd7c81037a3, State: Running, Role: LEADER
I20260927 12:35:44.111764 6024 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d14a038781486ab4968fd7c81037a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43489 } }
I20260927 12:35:44.112602 6019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:44.113332 6025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d14a038781486ab4968fd7c81037a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d14a038781486ab4968fd7c81037a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43489 } } }
I20260927 12:35:44.113444 6025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:44.113718 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.099207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.0.98.62:43489)
0927 12:35:44.099626 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.099643 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.099679 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:35:44.100390 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.100417 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.100796 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.101091 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.101977 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.102979 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.104202 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.104688 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.104807 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.107630 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.107636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.107650 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.107679 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.109770 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.110236 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.110243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.110250 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.110417 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.110724 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.110727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.110729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.111327 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.112208 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.112369 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.112681 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260927 12:35:44.115023 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d14a038781486ab4968fd7c81037a3. Latest consensus state: current_term: 1 leader_uuid: "c4d14a038781486ab4968fd7c81037a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d14a038781486ab4968fd7c81037a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43489 } } }
I20260927 12:35:44.115545 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3 [sys.catalog]: This master's current role is: LEADER
W20260927 12:35:44.117739 6040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:44.117841 6040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:44.117937 6031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:44.118737 6031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:44.125103 6031 catalog_manager.cc:1347] Generated new cluster ID: 17f9c87ca02249d0b9f8406010ac740c
I20260927 12:35:44.125185 6031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:44.138022 6031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:44.139025 6031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:44.143702 6031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d14a038781486ab4968fd7c81037a3: Generated new TSK 0
I20260927 12:35:44.144447 6031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:44.173365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43489
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:44.288411 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:44.288702 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:44.288745 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:44.292879 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:44.292951 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:44.293012 6045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:44.298430 6045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:44.299651 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:44.300732 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:44.312064 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:44.312073 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:44.312073 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:44.312757 6045 server_base.cc:1034] running on GCE node
I20260927 12:35:44.313195 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:44.313882 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:44.315037 6045 hybrid_clock.cc:648] HybridClock initialized: now 1790512544315009 us; error 49 us; skew 500 ppm
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.317713 6045 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:44.318608 6045 webserver.cc:460] Webserver started at http://127.0.98.1:44455/ using document root <none> and password file <none>
I20260927 12:35:44.319169 6045 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:44.319249 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:44.319450 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:44.321584 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "9aac9393da174dacb646b583b6b2d973"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "43071477dd2d6c63c97727c33a3ed5c0"
server_key_iv: "fb3e1a804f0fb1ab09616efa37678fe3"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.322124 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9aac9393da174dacb646b583b6b2d973"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "43071477dd2d6c63c97727c33a3ed5c0"
server_key_iv: "fb3e1a804f0fb1ab09616efa37678fe3"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.325979 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:44.328969 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.330320 6045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 12:35:44.330453 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "9aac9393da174dacb646b583b6b2d973"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "43071477dd2d6c63c97727c33a3ed5c0"
server_key_iv: "fb3e1a804f0fb1ab09616efa37678fe3"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.330566 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:44.345022 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:44.348048 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:44.348266 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:44.348843 6045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:44.349647 6045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:44.349695 6045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.349746 6045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:44.349772 6045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.358289 6045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42705
I20260927 12:35:44.358314 6173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42705 every 8 connection(s)
I20260927 12:35:44.359246 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:44.359941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6045
I20260927 12:35:44.360037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:44.360370 392 external_mini_cluster.cc:1428] Setting key 692d3e5df7074649e35d0de91014ffea
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:44.377238 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.361201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35047 (local address 127.0.98.62:43489)
0927 12:35:44.361328 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.361333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.363671 (+ 2338us) server_negotiation.cc:407] Connection header received
0927 12:35:44.365070 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.365074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.365131 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.365227 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.366469 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.367043 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.368931 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.369205 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.369321 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.373783 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.373787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.373789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.373816 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.375157 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.375688 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.375692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.375693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.375778 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.376143 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.376146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.376147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.376330 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.376415 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.376939 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.377068 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:44.378026 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.361662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43489 (local address 127.0.98.1:35047)
0927 12:35:44.363488 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:44.363537 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:44.364840 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:44.365420 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:44.365428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:44.365791 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:44.366320 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:44.366334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.367190 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.367193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:44.368685 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:44.368698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.369389 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.369394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:44.369631 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.370362 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:44.370409 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:44.373591 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:44.375276 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:44.375282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:44.375300 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:44.375581 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:44.375863 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:44.375866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:44.375868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:44.376020 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:44.376432 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:44.376439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:44.376694 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 12:35:44.376860 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 12:35:44.377194 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1615,"thread_start_us":124,"threads_started":1}
I20260927 12:35:44.379292 6174 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43489
I20260927 12:35:44.379528 6174 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:44.380054 6174 heartbeater.cc:511] Master 127.0.98.62:43489 requested a full tablet report, sending...
I20260927 12:35:44.381632 5978 ts_manager.cc:194] Registered new tserver with Master: 9aac9393da174dacb646b583b6b2d973 (127.0.98.1:42705)
I20260927 12:35:44.382937 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:44.388327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43489
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:44.497566 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:44.497845 6179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:44.497888 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:44.502054 6179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:44.502130 6179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:44.502192 6179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:44.507674 6179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:44.508772 6179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:44.509747 6179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:44.519737 6185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:44.519737 6184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:44.519742 6187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:44.520150 6179 server_base.cc:1034] running on GCE node
I20260927 12:35:44.520618 6179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:44.521416 6179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:44.522596 6179 hybrid_clock.cc:648] HybridClock initialized: now 1790512544522575 us; error 37 us; skew 500 ppm
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.525467 6179 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:44.526360 6179 webserver.cc:460] Webserver started at http://127.0.98.2:40735/ using document root <none> and password file <none>
I20260927 12:35:44.526885 6179 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:44.526945 6179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:44.527148 6179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:44.529247 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "d62223edfa1043d1af6d69ee2f6f051b"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "28abf21b193c1a606ae29fe79a820ed8"
server_key_iv: "4349fc7de1de0079af275b125428fe70"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.529793 6179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d62223edfa1043d1af6d69ee2f6f051b"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "28abf21b193c1a606ae29fe79a820ed8"
server_key_iv: "4349fc7de1de0079af275b125428fe70"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.533706 6179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:44.536685 6194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.538093 6179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:44.538219 6179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "d62223edfa1043d1af6d69ee2f6f051b"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "28abf21b193c1a606ae29fe79a820ed8"
server_key_iv: "4349fc7de1de0079af275b125428fe70"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.538352 6179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:44.554242 6179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:44.557233 6179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:44.557395 6179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:44.558007 6179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:44.558820 6179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:44.558866 6179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.558907 6179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:44.558936 6179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.568619 6179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46867
I20260927 12:35:44.568646 6307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46867 every 8 connection(s)
I20260927 12:35:44.569866 6179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:44.574697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6179
I20260927 12:35:44.574999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:44.575297 392 external_mini_cluster.cc:1428] Setting key 0281d8313316304a40c8b5cdb0a824f2
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:35:44.589061 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.572221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54085 (local address 127.0.98.62:43489)
0927 12:35:44.572367 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.572370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.573391 (+ 1021us) server_negotiation.cc:407] Connection header received
0927 12:35:44.575223 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.575227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.575289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.575346 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.577250 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.577765 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.579129 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.579449 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.579536 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.582643 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.582648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.582651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.582683 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.584358 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.585054 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.585059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.585061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.585155 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.587472 (+ 2317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.587477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.587479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.587758 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.587875 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.588715 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.588865 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:35:44.590251 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.572711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43489 (local address 127.0.98.2:54085)
0927 12:35:44.573232 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:44.573288 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:44.574949 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:44.575826 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:44.575836 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:44.576404 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:44.577077 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:44.577098 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.577888 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.577891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:44.578986 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:44.578998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.579753 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.579757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:44.579957 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.580618 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:44.580640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:44.582354 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:44.584510 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:44.584518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:44.584538 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:44.584924 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:44.587155 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:44.587158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:44.587159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:44.587336 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:44.587928 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:44.587937 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:44.588393 (+ 456us) client_negotiation.cc:769] Sending connection context
0927 12:35:44.588619 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 12:35:44.589036 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260927 12:35:44.591527 6308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43489
I20260927 12:35:44.591815 6308 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:44.592504 6308 heartbeater.cc:511] Master 127.0.98.62:43489 requested a full tablet report, sending...
I20260927 12:35:44.593868 5978 ts_manager.cc:194] Registered new tserver with Master: d62223edfa1043d1af6d69ee2f6f051b (127.0.98.2:46867)
I20260927 12:35:44.594640 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:44.607735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43489
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:44.721344 6313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:44.721617 6313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:44.721662 6313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:44.726100 6313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:44.726191 6313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:44.726266 6313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:44.732056 6313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:44.733414 6313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:44.734455 6313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:44.744787 6319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:44.744944 6313 server_base.cc:1034] running on GCE node
W20260927 12:35:44.744845 6321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:44.744848 6318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:44.745500 6313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:44.746176 6313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:44.747355 6313 hybrid_clock.cc:648] HybridClock initialized: now 1790512544747319 us; error 49 us; skew 500 ppm
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.750313 6313 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:44.751209 6313 webserver.cc:460] Webserver started at http://127.0.98.3:37733/ using document root <none> and password file <none>
I20260927 12:35:44.751735 6313 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:44.751804 6313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:44.752127 6313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:44.754252 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "a600b7930afd4d57b41bb9d05dc1f811"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "6cd27d0f7aac34056dd2ef7b807305e4"
server_key_iv: "a8ec96655fc520901684d3507b809f4f"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.754765 6313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a600b7930afd4d57b41bb9d05dc1f811"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "6cd27d0f7aac34056dd2ef7b807305e4"
server_key_iv: "a8ec96655fc520901684d3507b809f4f"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.758551 6313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:44.761077 6328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.762375 6313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:35:44.762478 6313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "a600b7930afd4d57b41bb9d05dc1f811"
format_stamp: "Formatted at 2026-09-27 12:35:44 on dist-test-slave-tzgx"
server_key: "6cd27d0f7aac34056dd2ef7b807305e4"
server_key_iv: "a8ec96655fc520901684d3507b809f4f"
server_key_version: "encryptionkey@0"
I20260927 12:35:44.762571 6313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:44.776659 6313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:44.779911 6313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:44.780082 6313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:44.780714 6313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:44.781536 6313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:44.781582 6313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.781621 6313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:44.781642 6313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:44.790597 6313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37521
I20260927 12:35:44.790683 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37521 every 8 connection(s)
I20260927 12:35:44.791497 6313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:44.794205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6313
I20260927 12:35:44.794325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:44.794637 392 external_mini_cluster.cc:1428] Setting key 46f8572550861e2f47f8c551aa592fce
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:44.805765 6022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.793273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57249 (local address 127.0.98.62:43489)
0927 12:35:44.793451 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.793456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.794390 (+ 934us) server_negotiation.cc:407] Connection header received
0927 12:35:44.796146 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.796149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.796200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.796283 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.797568 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.798121 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.799291 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.799506 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.799586 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.802185 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.802189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.802191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.802218 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.803578 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.804262 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.804265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.804267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.804353 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.804713 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.804717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.804718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.804924 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.805030 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.805531 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.805639 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:35:44.806640 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.793701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43489 (local address 127.0.98.3:57249)
0927 12:35:44.794204 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:44.794263 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:44.795883 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:44.796442 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:44.796449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:44.796904 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:44.797424 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:44.797439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.798246 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.798249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:44.799176 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:44.799186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.799630 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.799633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:44.799783 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.800415 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:44.800433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:44.801983 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:44.803740 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:44.803746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:44.803762 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:44.804154 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:44.804438 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:44.804440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:44.804441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:44.804600 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:44.805030 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:44.805035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:44.805311 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 12:35:44.805490 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 12:35:44.805786 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260927 12:35:44.808169 6442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43489
I20260927 12:35:44.808421 6442 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:44.808965 6442 heartbeater.cc:511] Master 127.0.98.62:43489 requested a full tablet report, sending...
I20260927 12:35:44.809984 5983 ts_manager.cc:194] Registered new tserver with Master: a600b7930afd4d57b41bb9d05dc1f811 (127.0.98.3:37521)
I20260927 12:35:44.810570 5983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57249
I20260927 12:35:44.820601 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/test-admin.keytab.
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.837200 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 12:35:44.839856 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:44.838531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.0.98.1:42705)
0927 12:35:44.838772 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.838786 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.838809 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:44.838898 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.838903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.839087 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.839205 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.839678 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44380: BlockingRecv error: recv got EOF from 127.0.0.1:44380 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
W20260927 12:35:44.840112 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.841130 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.841997 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:44.851408 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.842873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.98.1:42705)
0927 12:35:44.842995 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.842999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.843105 (+ 106us) server_negotiation.cc:407] Connection header received
0927 12:35:44.843238 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.843241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.843288 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.843351 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.843868 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.844710 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.845512 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.845900 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.846002 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.848190 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.848199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.848212 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.848238 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.849679 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.850108 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.850113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.850115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.850196 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.850454 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.850458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.850459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.850708 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.850925 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.851092 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.851240 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:35:44.855109 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:44.854139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.0.98.1:42705)
0927 12:35:44.854267 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.854271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.854386 (+ 115us) server_negotiation.cc:407] Connection header received
0927 12:35:44.854503 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.854506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.854545 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.854606 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.855017 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44386: BlockingRecv error: recv got EOF from 127.0.0.1:44386 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:35:44.855254 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.856328 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.857118 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:44.865439 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.857981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.0.98.1:42705)
0927 12:35:44.858091 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.858094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.858215 (+ 121us) server_negotiation.cc:407] Connection header received
0927 12:35:44.858307 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.858309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.858346 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.858411 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.858910 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.859438 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.860299 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.860536 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.860615 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.862693 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.862697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.862699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.862724 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.863894 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.864341 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.864344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.864345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.864420 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.864668 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.864671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.864673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.864842 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.864954 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.865181 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.865291 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:35:44.868073 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512512765972-392-0/krb5kdc/krb5cc' of type FILE
W20260927 12:35:44.868251 6449 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:44.867135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.98.1:42705)
0927 12:35:44.867284 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.867288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.867339 (+ 51us) server_negotiation.cc:407] Connection header received
0927 12:35:44.867451 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.867453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.867490 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.867552 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.868003 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44396: BlockingRecv error: recv got EOF from 127.0.0.1:44396 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.869169 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:44.870057 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[5938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:35:44.878439 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:44.870927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.0.98.1:42705)
0927 12:35:44.871046 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:44.871050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:44.871166 (+ 116us) server_negotiation.cc:407] Connection header received
0927 12:35:44.871251 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:44.871253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:44.871292 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:44.871356 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:44.871849 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.872424 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.873190 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:44.873418 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:44.873504 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:44.875590 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:44.875593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:44.875595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:44.875620 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:44.876962 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.877366 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.877369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.877370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.877464 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:44.877711 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:44.877714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:44.877715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:44.877892 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:44.878023 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:44.878180 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 12:35:44.878278 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:35:44.879491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6045
I20260927 12:35:44.884487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6179
I20260927 12:35:44.888880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6313
I20260927 12:35:44.895141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 5951
2026-09-27T12:35:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2078 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[6462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[6462](info): set up 2 sockets
Sep 27 12:35:44 dist-test-slave-tzgx krb5kdc[6462](info): commencing operation
krb5kdc: starting...
W20260927 12:35:45.856246 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:45 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 12:35:45.977048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:46.089530 6475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:46.089833 6475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:46.089876 6475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:46.094425 6475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:46.094532 6475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:46.094551 6475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:46.094568 6475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:46.094584 6475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:46.100404 6475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:46.101468 6475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:46.102443 6475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:46.110705 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.110702 6480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.110715 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:46.110931 6475 server_base.cc:1034] running on GCE node
I20260927 12:35:46.111634 6475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:46.112825 6475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:46.113981 6475 hybrid_clock.cc:648] HybridClock initialized: now 1790512546113962 us; error 33 us; skew 500 ppm
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:46.120663 6475 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 12:35:46.121742 6475 webserver.cc:460] Webserver started at http://127.0.98.62:40691/ using document root <none> and password file <none>
I20260927 12:35:46.122321 6475 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:46.122394 6475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:46.122601 6475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:46.124787 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "79cbd7058d314252ada6b7fe971c641c"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "4ce600e76d4cea3a98fea42de24d2a8e"
server_key_iv: "3de166099d8dfa8fd12407384921886a"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.125341 6475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "79cbd7058d314252ada6b7fe971c641c"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "4ce600e76d4cea3a98fea42de24d2a8e"
server_key_iv: "3de166099d8dfa8fd12407384921886a"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.129154 6475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:35:46.132016 6490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.133760 6475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:35:46.133901 6475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "79cbd7058d314252ada6b7fe971c641c"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "4ce600e76d4cea3a98fea42de24d2a8e"
server_key_iv: "3de166099d8dfa8fd12407384921886a"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.134033 6475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:46.148030 6475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:46.151441 6475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:46.151650 6475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:46.158608 6475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33875
I20260927 12:35:46.158896 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33875 every 8 connection(s)
I20260927 12:35:46.159595 6475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:46.162531 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:46.163589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6475
I20260927 12:35:46.163724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:46.163981 392 external_mini_cluster.cc:1428] Setting key 66cc2acd4766c010b2d48e07c86700a4
I20260927 12:35:46.169196 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Bootstrap starting.
I20260927 12:35:46.172035 6543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:46.173025 6543 log.cc:824] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:46.175220 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: No bootstrap required, opened a new log
I20260927 12:35:46.178782 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.165419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.0.98.62:33875)
0927 12:35:46.165830 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:46.165846 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:46.165881 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:35:46.166583 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:46.166603 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:46.166960 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:46.167263 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:46.168222 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.169255 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.170507 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.170995 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.171187 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.174162 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:46.174168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:46.174182 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:46.174211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:46.175856 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.176424 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.176429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.176434 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.176550 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.176828 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.176830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.176831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.177235 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:46.177397 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:46.177724 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 12:35:46.178023 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260927 12:35:46.179380 6543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } }
I20260927 12:35:46.179641 6543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:46.179702 6543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cbd7058d314252ada6b7fe971c641c, State: Initialized, Role: FOLLOWER
I20260927 12:35:46.180258 6543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } }
I20260927 12:35:46.180433 6543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:46.180501 6543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:46.180600 6543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:46.181764 6543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } }
I20260927 12:35:46.182108 6543 leader_election.cc:302] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cbd7058d314252ada6b7fe971c641c; no voters:
I20260927 12:35:46.182413 6543 leader_election.cc:288] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:46.182533 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:46.182746 6548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 LEADER]: Becoming Leader. State: Replica: 79cbd7058d314252ada6b7fe971c641c, State: Running, Role: LEADER
I20260927 12:35:46.183102 6548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } }
I20260927 12:35:46.183686 6543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:46.184681 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cbd7058d314252ada6b7fe971c641c. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } }
I20260927 12:35:46.184798 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:46.185176 6557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:46.185165 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } }
I20260927 12:35:46.185328 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:46.188796 6557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:46.195134 6557 catalog_manager.cc:1347] Generated new cluster ID: 21c7c51011ae44b1a1949b6e3297cf36
I20260927 12:35:46.195207 6557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:46.214775 6557 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:46.215806 6557 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:46.222720 6557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Generated new TSK 0
I20260927 12:35:46.223609 6557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 12:35:46.260030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33875
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:46.375087 6569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:46.375381 6569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:46.375425 6569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:46.379597 6569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:46.379684 6569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:46.379746 6569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:46.385293 6569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:46.386364 6569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:46.387360 6569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:46.396821 6577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.396831 6575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.396821 6574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:46.397035 6569 server_base.cc:1034] running on GCE node
I20260927 12:35:46.397519 6569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:46.398204 6569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:46.399394 6569 hybrid_clock.cc:648] HybridClock initialized: now 1790512546399376 us; error 30 us; skew 500 ppm
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:46.402086 6569 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:35:46.402981 6569 webserver.cc:460] Webserver started at http://127.0.98.1:39351/ using document root <none> and password file <none>
I20260927 12:35:46.403556 6569 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:46.403640 6569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:46.403861 6569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:46.405936 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "994b257ca30f4a6693158c2b3218bbf5"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "23a7c6ea2e37f8ba8a1a00095f79aefb"
server_key_iv: "bfce544d8f4748bb1aba90ae4100cf69"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.406531 6569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "994b257ca30f4a6693158c2b3218bbf5"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "23a7c6ea2e37f8ba8a1a00095f79aefb"
server_key_iv: "bfce544d8f4748bb1aba90ae4100cf69"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.410318 6569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:46.412863 6584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.414129 6569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:35:46.414247 6569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "994b257ca30f4a6693158c2b3218bbf5"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "23a7c6ea2e37f8ba8a1a00095f79aefb"
server_key_iv: "bfce544d8f4748bb1aba90ae4100cf69"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.414355 6569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:46.429329 6569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:46.432575 6569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:46.432771 6569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:46.433352 6569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:46.434224 6569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:46.434290 6569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.434346 6569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:46.434374 6569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.443591 6569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35845
I20260927 12:35:46.443621 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35845 every 8 connection(s)
I20260927 12:35:46.444561 6569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:46.446830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6569
I20260927 12:35:46.446970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:46.447265 392 external_mini_cluster.cc:1428] Setting key 098decc0041dd290a0302a23755384d1
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 12:35:46.460304 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.446352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52277 (local address 127.0.98.62:33875)
0927 12:35:46.446487 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:46.446490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:46.447421 (+ 931us) server_negotiation.cc:407] Connection header received
0927 12:35:46.448855 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:46.448857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:46.448901 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:46.448960 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:46.450485 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.451266 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.452601 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.452922 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.453032 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.456036 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:46.456040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:46.456042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:46.456075 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:46.457690 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.458440 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.458443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.458444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.458547 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.458933 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.458937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.458938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.459150 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:46.459240 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:46.459941 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 12:35:46.460129 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:35:46.461355 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.446721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:52277)
0927 12:35:46.447234 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:46.447293 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:46.448631 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:46.449157 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:46.449168 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:46.449654 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:46.450322 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:46.450356 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.451432 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.451435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:46.452473 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:46.452484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.453108 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.453112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:46.453395 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.454012 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:46.454030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:46.455679 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:46.457834 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:46.457841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:46.457889 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:46.458318 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:46.458635 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:46.458637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:46.458638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:46.458822 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:46.459244 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:46.459249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:46.459659 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 12:35:46.459881 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 12:35:46.460275 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260927 12:35:46.462795 6698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33875
I20260927 12:35:46.463084 6698 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:46.463716 6698 heartbeater.cc:511] Master 127.0.98.62:33875 requested a full tablet report, sending...
I20260927 12:35:46.465785 6507 ts_manager.cc:194] Registered new tserver with Master: 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845)
I20260927 12:35:46.467432 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 12:35:46.480376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33875
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:46.595242 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:46.595533 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:46.595578 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:46.599733 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:46.599804 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:46.599864 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:46.605327 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:46.606519 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:46.607563 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:46.616962 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.616977 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.616971 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:46.617278 6703 server_base.cc:1034] running on GCE node
I20260927 12:35:46.617692 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:46.618364 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:46.619554 6703 hybrid_clock.cc:648] HybridClock initialized: now 1790512546619532 us; error 37 us; skew 500 ppm
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:46.622318 6703 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:35:46.623245 6703 webserver.cc:460] Webserver started at http://127.0.98.2:40547/ using document root <none> and password file <none>
I20260927 12:35:46.623776 6703 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:46.623833 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:46.624001 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:46.626065 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "44e154ece3954fdb81c20f6f6afbbd94"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "538af44cd7831a8f5ae7ab629bfc6925"
server_key_iv: "76d7d04a314531b60582fb369d083744"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.626613 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44e154ece3954fdb81c20f6f6afbbd94"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "538af44cd7831a8f5ae7ab629bfc6925"
server_key_iv: "76d7d04a314531b60582fb369d083744"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.630419 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:46.632992 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.634313 6703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:35:46.634423 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "44e154ece3954fdb81c20f6f6afbbd94"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "538af44cd7831a8f5ae7ab629bfc6925"
server_key_iv: "76d7d04a314531b60582fb369d083744"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.634526 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:46.653388 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:46.656550 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:46.656729 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:46.657312 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:46.658095 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:46.658141 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.658179 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:46.658200 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.666708 6703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35521
I20260927 12:35:46.666777 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35521 every 8 connection(s)
I20260927 12:35:46.667620 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:46.677074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6703
I20260927 12:35:46.677173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:46.677461 392 external_mini_cluster.cc:1428] Setting key 79a0de66fda930a570cd8148b1d6430f
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:46.682053 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.669476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57613 (local address 127.0.98.62:33875)
0927 12:35:46.669610 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:46.669614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:46.670508 (+ 894us) server_negotiation.cc:407] Connection header received
0927 12:35:46.671950 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:46.671953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:46.672003 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:46.672083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:46.673320 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.673869 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.675121 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.675349 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.675443 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.678141 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:46.678145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:46.678148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:46.678180 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:46.679798 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.680405 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.680409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.680411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.680501 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.680897 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.680899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.680900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.681087 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:46.681158 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:46.681798 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 12:35:46.681900 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:35:46.683224 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.669876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:57613)
0927 12:35:46.670349 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:46.670393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:46.671678 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:46.672281 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:46.672288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:46.672656 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:46.673157 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:46.673171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.674028 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.674032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:46.674957 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:46.674967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.675542 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.675546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:46.675715 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.676395 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:46.676413 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:46.677946 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:46.679933 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:46.679938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:46.679954 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:46.680313 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:46.680589 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:46.680592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:46.680594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:46.680804 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:46.681176 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:46.681184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:46.681550 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 12:35:46.681766 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 12:35:46.682145 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260927 12:35:46.684459 6832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33875
I20260927 12:35:46.684696 6832 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:46.685247 6832 heartbeater.cc:511] Master 127.0.98.62:33875 requested a full tablet report, sending...
I20260927 12:35:46.686295 6507 ts_manager.cc:194] Registered new tserver with Master: 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521)
I20260927 12:35:46.687088 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57613
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:35:46.705338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33875
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:46.819464 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:46.819751 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:46.819809 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:46.824031 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:46.824136 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:46.824218 6837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:46.829722 6837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:46.830870 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:46.831918 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:46.842082 6845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.842170 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:46.842249 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:46.842475 6837 server_base.cc:1034] running on GCE node
I20260927 12:35:46.842902 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:46.843577 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:46.844762 6837 hybrid_clock.cc:648] HybridClock initialized: now 1790512546844740 us; error 34 us; skew 500 ppm
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:46.847647 6837 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:35:46.848663 6837 webserver.cc:460] Webserver started at http://127.0.98.3:37015/ using document root <none> and password file <none>
I20260927 12:35:46.849347 6837 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:46.849436 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:46.849644 6837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:46.851812 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "01de338cd64a4c45a45c8c21363c6183"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "650ec2cfc2b38c434d34478f36572558"
server_key_iv: "3fd96a7c702654e13d7a59222d0f8c2a"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.852424 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01de338cd64a4c45a45c8c21363c6183"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "650ec2cfc2b38c434d34478f36572558"
server_key_iv: "3fd96a7c702654e13d7a59222d0f8c2a"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.856410 6837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:46.858979 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.860410 6837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:46.860538 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "01de338cd64a4c45a45c8c21363c6183"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "650ec2cfc2b38c434d34478f36572558"
server_key_iv: "3fd96a7c702654e13d7a59222d0f8c2a"
server_key_version: "encryptionkey@0"
I20260927 12:35:46.860651 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:46.875771 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:46.878935 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:46.879125 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:46.879706 6837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:46.880595 6837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:46.880643 6837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.880704 6837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:46.880728 6837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:46.889428 6837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32803
I20260927 12:35:46.889469 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32803 every 8 connection(s)
I20260927 12:35:46.890312 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:46.892064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6837
I20260927 12:35:46.892203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:46.892500 392 external_mini_cluster.cc:1428] Setting key 4f24e8e5e899a669671e6da51c7d0f72
Sep 27 12:35:46 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:46.903867 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.892153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37781 (local address 127.0.98.62:33875)
0927 12:35:46.892298 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:46.892302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:46.893201 (+ 899us) server_negotiation.cc:407] Connection header received
0927 12:35:46.894551 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:46.894554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:46.894610 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:46.894684 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:46.895866 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.896442 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.897573 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.897787 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.897874 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.900488 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:46.900491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:46.900494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:46.900521 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:46.901855 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.902385 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.902388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.902389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.902467 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:46.902786 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:46.902789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:46.902791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:46.902972 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:46.903057 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:46.903581 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 12:35:46.903703 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:35:46.904799 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:46.892573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:37781)
0927 12:35:46.893053 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:46.893097 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:46.894353 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:46.894860 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:46.894867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:46.895234 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:46.895732 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:46.895746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.896558 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.896561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:46.897457 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:46.897467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:46.897937 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:46.897941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:46.898099 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:46.898677 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:46.898693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:46.900295 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:46.901991 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:46.901997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:46.902013 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:46.902279 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:46.902553 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:46.902556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:46.902557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:46.902699 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:46.903071 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:46.903080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:46.903347 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 12:35:46.903525 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 12:35:46.903870 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260927 12:35:46.905805 6966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33875
I20260927 12:35:46.906056 6966 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:46.906580 6966 heartbeater.cc:511] Master 127.0.98.62:33875 requested a full tablet report, sending...
I20260927 12:35:46.907531 6506 ts_manager.cc:194] Registered new tserver with Master: 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3:32803)
I20260927 12:35:46.908192 6506 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37781
I20260927 12:35:46.918720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 12:35:46.943521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43279
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33875,127.0.98.61:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:47.054960 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:47.055246 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:47.055286 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:47.059401 6971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:47.059470 6971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:47.059489 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:47.059504 6971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:47.059520 6971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:47.065091 6971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33875,127.0.98.61:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
W20260927 12:35:47.065320 6971 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33875,127.0.98.61:43279'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 12:35:47.066190 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:47.067149 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:47.075695 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.075716 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.075699 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.075793 6971 server_base.cc:1034] running on GCE node
I20260927 12:35:47.076656 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:47.077839 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:47.079027 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790512547079005 us; error 38 us; skew 500 ppm
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:47.081837 6971 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 12:35:47.082856 6971 webserver.cc:460] Webserver started at http://127.0.98.61:37993/ using document root <none> and password file <none>
I20260927 12:35:47.083413 6971 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.083480 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.083668 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:47.085973 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/instance:
uuid: "c77fa177dd194f87bb762f1777a84b9a"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "a2c5397560ad62f8fdde5fe5ac950c18"
server_key_iv: "8e7feaa6fb5fe0a04b0b8da306230b49"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.086555 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal/instance:
uuid: "c77fa177dd194f87bb762f1777a84b9a"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "a2c5397560ad62f8fdde5fe5ac950c18"
server_key_iv: "8e7feaa6fb5fe0a04b0b8da306230b49"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.090409 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 12:35:47.093014 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.094310 6971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:35:47.094435 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
uuid: "c77fa177dd194f87bb762f1777a84b9a"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "a2c5397560ad62f8fdde5fe5ac950c18"
server_key_iv: "8e7feaa6fb5fe0a04b0b8da306230b49"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.094548 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.108256 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:47.111483 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:47.111667 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:47.118319 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43279
I20260927 12:35:47.118634 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43279 every 8 connection(s)
I20260927 12:35:47.119299 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:35:47.122154 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:47.125962 7039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:35:47.129590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 6971
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:35:47.140637 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.127867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.0.98.62:33875)
0927 12:35:47.128051 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.128056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.128926 (+ 870us) server_negotiation.cc:407] Connection header received
0927 12:35:47.129980 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.129983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.130031 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.130110 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.131690 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.132427 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.133726 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.133955 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.134044 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.136539 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.136543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.136545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.136570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.138108 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.138674 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.138678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.138680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.138771 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.139141 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.139144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.139146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.139459 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.139558 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.140042 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.140486 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 12:35:47.141316 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.130448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.0.98.61:43279)
0927 12:35:47.130716 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.130727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.130750 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:47.130898 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.130902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.131066 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.131188 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.132139 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.133142 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.133933 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.134272 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.134366 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.136921 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.136927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.136938 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.136961 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.138653 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.139086 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.139091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.139093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.139194 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.139459 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.139462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.139464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.139807 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.139959 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.140234 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.140448 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":117,"threads_started":1}
I20260927 12:35:47.141800 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.128285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:46706)
0927 12:35:47.128761 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.128819 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.129761 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.130369 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.130379 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.130892 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.131507 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.131529 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.132593 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.132597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.133586 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.133596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.134094 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.134098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.134256 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.134812 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.134833 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.136369 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.138253 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.138261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.138272 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.138582 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.138876 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.138879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.138881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.139050 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.139559 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.139564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.139814 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.139978 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.141671 (+ 1693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1417,"thread_start_us":123,"threads_started":1}
I20260927 12:35:47.143033 7039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:35:47.151870 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.143531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56888)
0927 12:35:47.144420 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.144435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.144529 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.144782 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.144785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.144940 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.145037 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.145043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.145798 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.145801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.146327 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.146334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.146833 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.146836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.146923 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.147387 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.147399 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.148728 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.150415 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.150417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.150419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.150629 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.150950 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.150952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.150954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.151058 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.151387 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.151389 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.151472 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.151557 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.151679 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795}
I20260927 12:35:47.151885 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.143794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.0.98.61:43279)
0927 12:35:47.143903 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.143909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.144495 (+ 586us) server_negotiation.cc:407] Connection header received
0927 12:35:47.144623 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.144625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.144664 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.144731 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.145155 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.145686 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.146447 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.146702 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.146788 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.148898 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.148901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.148904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.148925 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.150257 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.150738 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.150741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.150743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.150854 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.151136 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.151138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.151139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.151309 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.151391 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.151597 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.151737 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":32}
I20260927 12:35:47.155200 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Bootstrap starting.
I20260927 12:35:47.157578 7039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:47.158300 7039 log.cc:824] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:47.160183 7039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: No bootstrap required, opened a new log
I20260927 12:35:47.163184 7039 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } }
I20260927 12:35:47.163403 7039 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:47.163460 7039 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77fa177dd194f87bb762f1777a84b9a, State: Initialized, Role: FOLLOWER
I20260927 12:35:47.163919 7039 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } }
I20260927 12:35:47.164485 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } } }
I20260927 12:35:47.164618 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:47.165239 7039 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:35:47.170346 7056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:47.170441 7056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:47.171820 6971 master_runner.cc:428] Detected that this master c77fa177dd194f87bb762f1777a84b9a is joining an existing cluster
I20260927 12:35:47.171900 6971 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43279
I20260927 12:35:47.183427 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.174539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.0.98.62:33875)
0927 12:35:47.174651 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.174657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.174967 (+ 310us) server_negotiation.cc:407] Connection header received
0927 12:35:47.175049 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.175053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.175109 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.175205 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.176214 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.176764 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.177970 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.178326 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.178439 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.179779 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.179783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.179785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.179809 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.181364 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182088 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.182092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.182095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.182194 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182490 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.182494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.182496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.182713 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.182810 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.183127 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.183257 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 12:35:47.183466 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.174424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.0.98.61:43279)
0927 12:35:47.174661 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.174666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.174927 (+ 261us) server_negotiation.cc:407] Connection header received
0927 12:35:47.175052 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.175055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.175206 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.175270 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.175937 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.176747 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.178558 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.178909 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.179010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.180585 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.180590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.180593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.180616 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.182052 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182376 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.182380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.182382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.182445 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182673 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.182677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.182678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.182866 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.182944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.183174 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.183330 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":180}
I20260927 12:35:47.183480 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.174306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56898)
0927 12:35:47.174827 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.174842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.174953 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.175398 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.175400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.175625 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.175817 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.175822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.176865 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.176868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.177844 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.177853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.179012 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.179015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.179086 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.179983 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.179997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.180421 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.182128 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.182133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.182297 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.182520 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.182525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.182599 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.182945 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.182948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.183044 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.183124 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.183277 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":94,"spinlock_wait_cycles":768,"thread_start_us":113,"threads_started":1}
I20260927 12:35:47.184217 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.174302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:46718)
0927 12:35:47.174827 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.174842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.174970 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.175253 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.175257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.175592 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.175812 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.175821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.176916 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.176919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.177851 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.177857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.178460 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.178463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.178529 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.179279 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.179293 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.179646 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.181611 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.181615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.181617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.181987 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.182302 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.182305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.182307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.182395 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.182800 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.182803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.182903 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.183599 (+ 696us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.183933 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":81,"thread_start_us":144,"threads_started":1}
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:35:47.198961 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.189552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56914)
0927 12:35:47.189888 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.189933 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.190073 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.190370 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.190376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.190770 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.190931 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.190938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.191658 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.191660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.192381 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.192392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.192810 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.192814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.192936 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.193449 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.193461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.195021 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.197042 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.197054 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.197061 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.197404 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.197685 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.197688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.197691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.197786 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.198225 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.198233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.198388 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.198544 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.198769 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260927 12:35:47.199074 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.189579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.0.98.61:43279)
0927 12:35:47.189697 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.189702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.190012 (+ 310us) server_negotiation.cc:407] Connection header received
0927 12:35:47.190200 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.190204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.190255 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.190321 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.191045 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.191543 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.192489 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.192710 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.192796 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.195162 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.195166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.195169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.195196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.196920 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.197508 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.197511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.197513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.197602 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.197879 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.197883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.197885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.198126 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.198217 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.198604 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.198728 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:35:47.200683 6506 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43279: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43279
}
attrs {
promote: true
}
}
dest_uuid: "79cbd7058d314252ada6b7fe971c641c"
cas_config_opid_index: -1
I20260927 12:35:47.201469 6506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 12:35:47.203718 7067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c77fa177dd194f87bb762f1777a84b9a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } }
I20260927 12:35:47.204540 7013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:47.206233 7067 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 12:35:47.206584 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } }
I20260927 12:35:47.206682 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:47.207053 6971 master.cc:537] Master@127.0.98.61:43279 shutting down...
I20260927 12:35:47.207468 7013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Refusing update from remote peer 79cbd7058d314252ada6b7fe971c641c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 12:35:47.208206 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:35:47.210243 6491 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 12:35:47.211201 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c -> Peer c77fa177dd194f87bb762f1777a84b9a (127.0.98.61:43279): Couldn't send request to peer c77fa177dd194f87bb762f1777a84b9a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 12:35:47.211930 6971 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:47.212280 6971 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:47.212409 6971 tablet_replica.cc:330] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: stopping tablet replica
I20260927 12:35:47.218485 6971 master.cc:552] Master@127.0.98.61:43279 shutdown complete.
I20260927 12:35:47.218541 6971 master_runner.cc:305] Clearing existing system tablet
I20260927 12:35:47.218762 6971 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.218804 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.219974 6971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.220806 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.221035 6971 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.221144 6971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.221174 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
uuid: "c77fa177dd194f87bb762f1777a84b9a"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "a2c5397560ad62f8fdde5fe5ac950c18"
server_key_iv: "8e7feaa6fb5fe0a04b0b8da306230b49"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.223166 6971 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 12:35:47.225014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40475
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:35:47.225291 6971 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 12:35:47.225389 6971 log.cc:1196] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 12:35:47.225606 6971 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Deleting consensus metadata
I20260927 12:35:47.225752 6971 master_runner.cc:315] Copying system tablet from 127.0.98.62:33875
I20260927 12:35:47.226823 6971 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33875
I20260927 12:35:47.234166 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.227449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.0.98.62:33875)
0927 12:35:47.227573 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.227577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.227740 (+ 163us) server_negotiation.cc:407] Connection header received
0927 12:35:47.227852 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.227855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.227902 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.227969 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.228551 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.229107 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.229945 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.230162 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.230246 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.231353 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.231357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.231360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.231393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.232682 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.233073 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.233077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.233079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.233152 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.233396 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.233399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.233401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.233572 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.233657 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.233901 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.234006 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:35:47.234184 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.227354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:46726)
0927 12:35:47.227654 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.227667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.227772 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.227994 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.227997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.228325 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.228429 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.228436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.229216 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.229219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.229818 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.229829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.230261 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.230265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.230356 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.230889 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.230902 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.231213 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.232782 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.232786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.232788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.232987 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.233230 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.233233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.233235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.233314 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.233658 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.233661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.233755 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.233839 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.233999 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":80,"threads_started":1}
I20260927 12:35:47.235002 6527 tablet_copy_service.cc:140] P 79cbd7058d314252ada6b7fe971c641c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c77fa177dd194f87bb762f1777a84b9a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46726)
I20260927 12:35:47.235163 6527 tablet_copy_service.cc:161] P 79cbd7058d314252ada6b7fe971c641c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c77fa177dd194f87bb762f1777a84b9a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46726: session id = c77fa177dd194f87bb762f1777a84b9a-00000000000000000000000000000000
I20260927 12:35:47.236665 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 12:35:47.238237 6971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 12:35:47.240363 6971 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: tablet copy: Starting download of 0 data blocks...
I20260927 12:35:47.240538 6971 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: tablet copy: Starting download of 1 WAL segments...
I20260927 12:35:47.242063 6971 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 12:35:47.243577 6527 tablet_copy_service.cc:342] P 79cbd7058d314252ada6b7fe971c641c: Request end of tablet copy session c77fa177dd194f87bb762f1777a84b9a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46726
I20260927 12:35:47.243681 6527 tablet_copy_service.cc:433] P 79cbd7058d314252ada6b7fe971c641c: ending tablet copy session c77fa177dd194f87bb762f1777a84b9a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c77fa177dd194f87bb762f1777a84b9a
I20260927 12:35:47.250216 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:47.251971 7083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.252264 7081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.252349 6971 server_base.cc:1034] running on GCE node
W20260927 12:35:47.252816 7080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.253069 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:47.253355 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:47.254494 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790512547254480 us; error 30 us; skew 500 ppm
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:47.255882 6971 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 12:35:47.256356 6971 webserver.cc:460] Webserver started at http://127.0.98.61:40973/ using document root <none> and password file <none>
I20260927 12:35:47.256536 6971 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.256569 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.257624 6971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.258466 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.258688 6971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.258754 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
uuid: "c77fa177dd194f87bb762f1777a84b9a"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "a2c5397560ad62f8fdde5fe5ac950c18"
server_key_iv: "8e7feaa6fb5fe0a04b0b8da306230b49"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.258836 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.270457 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:47.272469 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:47.276943 6971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43279
I20260927 12:35:47.277206 7142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43279 every 8 connection(s)
I20260927 12:35:47.277526 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:35:47.278945 7143 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:35:47.279906 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Bootstrap starting.
I20260927 12:35:47.288591 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:35:47.288851 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: Bootstrap complete.
I20260927 12:35:47.289316 7143 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } }
I20260927 12:35:47.289438 7143 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c77fa177dd194f87bb762f1777a84b9a, State: Initialized, Role: LEARNER
I20260927 12:35:47.289535 7143 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } }
I20260927 12:35:47.289839 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } }
I20260927 12:35:47.290066 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: LEARNER
I20260927 12:35:47.289990 7143 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:47.293254 7157 catalog_manager.cc:1259] Loaded cluster ID: 21c7c51011ae44b1a1949b6e3297cf36
I20260927 12:35:47.293304 7157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: loading cluster ID for follower catalog manager: success
I20260927 12:35:47.294780 7157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: acquiring CA information for follower catalog manager: success
I20260927 12:35:47.295888 7157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 12:35:47.369649 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:47.369977 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:47.370018 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:47.374269 7074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:47.374366 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:47.374387 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:47.374403 7074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:47.374418 7074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:47.380214 7074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:47.381312 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:47.382365 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:47.391580 7162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.391657 7074 server_base.cc:1034] running on GCE node
W20260927 12:35:47.391541 7163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.391798 7165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.392418 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:47.393608 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:47.394797 7074 hybrid_clock.cc:648] HybridClock initialized: now 1790512547394776 us; error 36 us; skew 500 ppm
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:47.397825 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 12:35:47.398789 7074 webserver.cc:460] Webserver started at http://127.0.98.60:33311/ using document root <none> and password file <none>
I20260927 12:35:47.399346 7074 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.399415 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.399610 7074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:47.401840 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/instance:
uuid: "ad26acff36da44449acd6536d016bd9e"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "6efcaaf915492695ef41cc0d1468e4eb"
server_key_iv: "dd31bc50acb219b118dc62be13e436ba"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.402436 7074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal/instance:
uuid: "ad26acff36da44449acd6536d016bd9e"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "6efcaaf915492695ef41cc0d1468e4eb"
server_key_iv: "dd31bc50acb219b118dc62be13e436ba"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.406641 7074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:47.409684 7172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.411084 7074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:47.411216 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
uuid: "ad26acff36da44449acd6536d016bd9e"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "6efcaaf915492695ef41cc0d1468e4eb"
server_key_iv: "dd31bc50acb219b118dc62be13e436ba"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.411363 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.442008 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:47.445626 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:47.445797 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:47.453114 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40475
I20260927 12:35:47.453409 7224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40475 every 8 connection(s)
I20260927 12:35:47.454106 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:35:47.457216 7225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:47.461102 7225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:35:47.463034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 7074
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:47.470239 6698 heartbeater.cc:503] Master 127.0.98.62:33875 was elected leader, sending a full tablet report...
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:35:47.474563 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.463098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.0.98.62:33875)
0927 12:35:47.463255 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.463260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.464073 (+ 813us) server_negotiation.cc:407] Connection header received
0927 12:35:47.465111 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.465115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.465169 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.465259 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.466165 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.466722 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.467738 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.467958 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.468047 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.470699 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.470704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.470706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.470736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.472308 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.472875 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.472879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.472881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.472968 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.473366 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.473370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.473371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.473609 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.473681 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.474262 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.474403 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:35:47.475389 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.463861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.98.60:40475)
0927 12:35:47.464314 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.464323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.464356 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:35:47.464918 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.464944 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.465161 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.465430 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.466285 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.467152 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.468225 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.468599 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.468695 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.471078 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.471086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.471102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.471131 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.472944 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.473396 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.473402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.473403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.473478 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.473721 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.473724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.473726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.473993 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.474144 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.474370 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.474558 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":142,"threads_started":1}
I20260927 12:35:47.475888 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.463344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:46740)
0927 12:35:47.463915 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.463976 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.464908 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.465333 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.465343 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.465684 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.466024 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.466037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.466852 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.466855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.467620 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.467628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.468128 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.468133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.468307 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.468883 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.468899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.470510 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.472440 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.472449 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.472462 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.472779 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.473052 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.473055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.473057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.473255 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.473707 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.473714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.474029 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.474210 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.475744 (+ 1534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":1218,"spinlock_wait_cycles":7808,"thread_start_us":151,"threads_started":1}
I20260927 12:35:47.476982 7225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:35:47.485402 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.477484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56922)
0927 12:35:47.477585 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.477599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.477666 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.478127 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.478130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.478281 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.478383 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.478388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.479250 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.479254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.479795 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.479803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.480258 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.480261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.480366 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.480807 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.480819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.482011 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.483610 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.483614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.483616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.483866 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.484302 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.484306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.484308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.484404 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.484858 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.484861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.484968 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.485062 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.485221 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:35:47.485404 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.477694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.98.61:43279)
0927 12:35:47.477935 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.477939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.477950 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:35:47.477983 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.477985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.478023 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.478099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.478491 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.479151 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.479899 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.480164 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.480257 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.482145 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.482148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.482150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.482175 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.483499 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.483955 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.483958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.483959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.484040 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.484498 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.484502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.484504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.484701 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.484826 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.485125 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.485245 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260927 12:35:47.486260 7225 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:35:47.494202 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.486763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38334)
0927 12:35:47.486860 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.486871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.486943 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.487304 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.487306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.487463 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.487542 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.487547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.488457 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.488460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.488973 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.488981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.489470 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.489473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.489535 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.490015 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.490030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.491330 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.493005 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.493008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.493009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.493191 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.493402 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.493404 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.493405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.493473 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.493772 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.493775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.493854 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.493944 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.494058 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 12:35:47.494231 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.486883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.0.98.60:40475)
0927 12:35:47.487097 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.487101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.487115 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:35:47.487144 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.487146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.487181 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.487254 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.487675 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.488335 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.489089 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.489343 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.489431 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.491475 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.491478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.491480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.491501 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.492904 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.493265 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.493268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.493269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.493330 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.493530 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.493532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.493533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.493695 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.493761 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.493982 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.494113 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":54}
I20260927 12:35:47.497818 7225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Bootstrap starting.
I20260927 12:35:47.500255 7225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:47.501438 7225 log.cc:824] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:47.503517 7225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: No bootstrap required, opened a new log
I20260927 12:35:47.506922 7225 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } }
I20260927 12:35:47.507208 7225 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:47.507257 7225 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad26acff36da44449acd6536d016bd9e, State: Initialized, Role: FOLLOWER
I20260927 12:35:47.507990 7225 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } }
I20260927 12:35:47.508658 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } } }
I20260927 12:35:47.508816 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:47.509383 7225 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:35:47.514047 7242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:47.514159 7242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:47.517678 7074 master_runner.cc:428] Detected that this master ad26acff36da44449acd6536d016bd9e is joining an existing cluster
I20260927 12:35:47.517755 7074 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40475
I20260927 12:35:47.529129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6569
I20260927 12:35:47.531390 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.520242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.0.98.61:43279)
0927 12:35:47.520358 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.520363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.520832 (+ 469us) server_negotiation.cc:407] Connection header received
0927 12:35:47.521007 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.521010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.521063 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.521115 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.521906 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.522731 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525647 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.526011 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.526087 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.527419 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.527422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.527424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.527453 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.528977 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.530096 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.530100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.530102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.530197 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.530468 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.530471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.530473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.530696 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.530819 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.531103 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.531234 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:47.531410 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.520448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56924)
0927 12:35:47.520734 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.520748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.520898 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.521254 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.521257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.521594 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.521766 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.521773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.522868 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.522871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.525503 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.525514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.526237 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.526240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.526345 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.526771 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.526784 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.527252 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.529109 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.529113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.529115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.529974 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.530285 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.530288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.530290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.530387 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.530794 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.530797 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.530923 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.531042 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.531197 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":150,"spinlock_wait_cycles":5120,"thread_start_us":94,"threads_started":1}
I20260927 12:35:47.535966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35845
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33875
--webserver_port=39351
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:35:47.561255 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.520810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.0.98.60:40475)
0927 12:35:47.520897 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.520901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.521322 (+ 421us) server_negotiation.cc:407] Connection header received
0927 12:35:47.521439 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.521442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.521477 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.521519 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.522021 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.522718 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525030 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.525475 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525552 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.557862 (+ 32310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.557866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.557868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.557890 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.559420 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.560016 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.560020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.560021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.560149 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.560391 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.560394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.560396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.560609 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.560673 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.560931 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.561093 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":36}
I20260927 12:35:47.561453 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.520239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.0.98.62:33875)
0927 12:35:47.520368 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.520373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.520763 (+ 390us) server_negotiation.cc:407] Connection header received
0927 12:35:47.520967 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.520970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.521041 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.521100 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.521886 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.522723 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525042 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.525422 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525509 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.557199 (+ 31690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.557211 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.557215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.557259 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.559324 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.559828 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.559831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.559833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.559918 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.560258 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.560261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.560263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.560501 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.560587 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.561172 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.561285 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:35:47.561635 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.520182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:46752)
0927 12:35:47.520685 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.520699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.520850 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.521249 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.521251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.521601 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.521757 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.521764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.522834 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.522837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.524877 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.524891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.525676 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.525803 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.526366 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.526386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.556597 (+ 30211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.559480 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.559483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.559486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.559719 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.560012 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.560015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.560017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.560160 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.560628 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.560631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.560731 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.560859 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.561030 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":732,"spinlock_wait_cycles":6272,"thread_start_us":127,"threads_started":1}
I20260927 12:35:47.562373 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.520967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38346)
0927 12:35:47.521258 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.521270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.521364 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.521648 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.521650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.521826 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.521921 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.521927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.522834 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.522837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.524886 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.524892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.525676 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.525680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.525807 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.557249 (+ 31442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.557266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.557705 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.559652 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.559655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.559657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.559920 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.560229 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.560231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.560233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.560316 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.560725 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.560727 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.560795 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.560990 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.562193 (+ 1203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":918,"thread_start_us":107,"threads_started":1}
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512546, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:35:47.575259 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.565750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38348)
0927 12:35:47.565876 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.565891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.565977 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.566269 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.566271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.566483 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.566591 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.566597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.567608 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.567612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.568533 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.568545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.569111 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.569115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.569212 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.569832 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.569850 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.571508 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.573586 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.573590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.573592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.573856 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.574165 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.574168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.574170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.574263 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.574697 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.574700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.574807 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.574912 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.575080 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 12:35:47.575258 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.565810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.0.98.60:40475)
0927 12:35:47.565949 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.565954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.565971 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:35:47.566090 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.566094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.566153 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.566239 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.566698 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.567480 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.568662 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.568992 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.569112 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.571680 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.571684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.571687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.571721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.573455 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.573978 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.573981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.573983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.574080 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.574347 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.574350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.574351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.574599 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.574702 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.574973 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.575099 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:35:47.577047 6506 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40475: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ad26acff36da44449acd6536d016bd9e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40475
}
attrs {
promote: true
}
}
dest_uuid: "79cbd7058d314252ada6b7fe971c641c"
cas_config_opid_index: 5
I20260927 12:35:47.577673 6506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } }
I20260927 12:35:47.578742 7067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ad26acff36da44449acd6536d016bd9e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.579864 7199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:47.580271 7067 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40475
I20260927 12:35:47.580461 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.580554 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:47.581278 7074 master.cc:537] Master@127.0.98.60:40475 shutting down...
I20260927 12:35:47.582892 7199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Refusing update from remote peer 79cbd7058d314252ada6b7fe971c641c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 12:35:47.584204 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:35:47.586453 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c -> Peer ad26acff36da44449acd6536d016bd9e (127.0.98.60:40475): Couldn't send request to peer ad26acff36da44449acd6536d016bd9e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 12:35:47.589265 7074 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:47.589593 7074 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:47.589704 7074 tablet_replica.cc:330] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: stopping tablet replica
I20260927 12:35:47.591961 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.579226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.0.98.61:43279)
0927 12:35:47.581706 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.581711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.586705 (+ 4994us) server_negotiation.cc:407] Connection header received
0927 12:35:47.586902 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.586905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.586965 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.587022 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:47.587422 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.588455 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.590182 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.590816 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.590933 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.591102 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.591532 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.591628 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351}
I20260927 12:35:47.592407 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.579000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56940)
0927 12:35:47.586627 (+ 7627us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.586642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.586781 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.587173 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.587176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.587195 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:47.587292 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.587298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.588607 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.588610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.589975 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.589988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.591251 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.591255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.591353 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.591408 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.592191 (+ 783us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.592285 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7543}
I20260927 12:35:47.592978 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 LEARNER]: Refusing update from remote peer 79cbd7058d314252ada6b7fe971c641c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 12:35:47.593653 7066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 12:35:47.595602 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ad26acff36da44449acd6536d016bd9e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.597937 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cbd7058d314252ada6b7fe971c641c. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.598040 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: LEARNER
I20260927 12:35:47.598127 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: true } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.598196 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: LEARNER
I20260927 12:35:47.599047 7066 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: attempting to promote NON_VOTER c77fa177dd194f87bb762f1777a84b9a to VOTER
I20260927 12:35:47.599189 7074 master.cc:552] Master@127.0.98.60:40475 shutdown complete.
I20260927 12:35:47.599227 7074 master_runner.cc:305] Clearing existing system tablet
I20260927 12:35:47.599486 7074 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.599536 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.599540 7066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } }
I20260927 12:35:47.600955 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.601593 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 LEARNER]: Refusing update from remote peer 79cbd7058d314252ada6b7fe971c641c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 12:35:47.601997 7067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 12:35:47.602351 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.600981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38362)
0927 12:35:47.602077 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.602151 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":88,"threads_started":1}
I20260927 12:35:47.603868 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.602744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38366)
0927 12:35:47.603735 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.603795 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931}
W20260927 12:35:47.603961 6491 consensus_peers.cc:487] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c -> Peer ad26acff36da44449acd6536d016bd9e (127.0.98.60:40475): Couldn't send request to peer ad26acff36da44449acd6536d016bd9e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:35:47.604401 7066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c77fa177dd194f87bb762f1777a84b9a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.604888 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c77fa177dd194f87bb762f1777a84b9a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.606323 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.606446 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:47.606451 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.606549 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:47.606571 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.606638 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:47.607236 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.607450 7074 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.607589 7074 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 12:35:47.607628 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
uuid: "ad26acff36da44449acd6536d016bd9e"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "6efcaaf915492695ef41cc0d1468e4eb"
server_key_iv: "dd31bc50acb219b118dc62be13e436ba"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.609463 7074 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:47.611482 7074 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 12:35:47.611564 7074 log.cc:1196] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 12:35:47.611788 7074 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Deleting consensus metadata
I20260927 12:35:47.611945 7074 master_runner.cc:315] Copying system tablet from 127.0.98.62:33875
I20260927 12:35:47.612985 7074 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33875
I20260927 12:35:47.620323 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.613572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.98.62:33875)
0927 12:35:47.613698 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.613702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.613910 (+ 208us) server_negotiation.cc:407] Connection header received
0927 12:35:47.614021 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.614024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.614072 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.614139 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.614592 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.615184 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.616038 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.616324 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.616409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.617455 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.617459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.617462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.617487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.618774 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.619177 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.619180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.619182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.619258 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.619499 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.619502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.619504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.619681 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.619764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.620003 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.620192 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:35:47.620323 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.613510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:46760)
0927 12:35:47.613813 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.613825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.613940 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.614160 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.614163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.614370 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.614461 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.614466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.615289 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.615292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.615930 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.615939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.616435 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.616439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.616533 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.617002 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.617017 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.617334 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.618881 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.618884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.618887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.619093 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.619334 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.619346 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.619348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.619425 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.619763 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.619766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.619861 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.619944 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.620139 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":84,"threads_started":1}
I20260927 12:35:47.621088 6527 tablet_copy_service.cc:140] P 79cbd7058d314252ada6b7fe971c641c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ad26acff36da44449acd6536d016bd9e ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46760)
I20260927 12:35:47.621176 6527 tablet_copy_service.cc:161] P 79cbd7058d314252ada6b7fe971c641c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ad26acff36da44449acd6536d016bd9e at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46760: session id = ad26acff36da44449acd6536d016bd9e-00000000000000000000000000000000
I20260927 12:35:47.621987 6527 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 12:35:47.623059 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 12:35:47.625039 7074 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: tablet copy: Starting download of 0 data blocks...
I20260927 12:35:47.625207 7074 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: tablet copy: Starting download of 1 WAL segments...
I20260927 12:35:47.626538 7074 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 12:35:47.628139 6527 tablet_copy_service.cc:342] P 79cbd7058d314252ada6b7fe971c641c: Request end of tablet copy session ad26acff36da44449acd6536d016bd9e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46760
I20260927 12:35:47.628224 6527 tablet_copy_service.cc:433] P 79cbd7058d314252ada6b7fe971c641c: ending tablet copy session ad26acff36da44449acd6536d016bd9e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ad26acff36da44449acd6536d016bd9e
I20260927 12:35:47.634965 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:47.636670 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.636682 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.636996 7265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.640295 7074 server_base.cc:1034] running on GCE node
I20260927 12:35:47.640512 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:47.640825 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:47.641968 7074 hybrid_clock.cc:648] HybridClock initialized: now 1790512547641959 us; error 37 us; skew 500 ppm
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:47.643347 7074 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 12:35:47.643836 7074 webserver.cc:460] Webserver started at http://127.0.98.60:37879/ using document root <none> and password file <none>
I20260927 12:35:47.644023 7074 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.644057 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.645171 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.646062 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.646268 7074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:47.646337 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
uuid: "ad26acff36da44449acd6536d016bd9e"
format_stamp: "Formatted at 2026-09-27 12:35:47 on dist-test-slave-tzgx"
server_key: "6efcaaf915492695ef41cc0d1468e4eb"
server_key_iv: "dd31bc50acb219b118dc62be13e436ba"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.646420 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.656668 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:47.658463 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:47.663448 7074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40475
I20260927 12:35:47.663468 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40475 every 8 connection(s)
I20260927 12:35:47.664003 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:35:47.666077 7327 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:35:47.667138 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Bootstrap starting.
I20260927 12:35:47.677541 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:35:47.677834 7327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Bootstrap complete.
I20260927 12:35:47.678356 7327 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } }
I20260927 12:35:47.678470 7327 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad26acff36da44449acd6536d016bd9e, State: Initialized, Role: LEARNER
I20260927 12:35:47.678566 7327 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } }
I20260927 12:35:47.678900 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:47.679040 7327 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:47.679072 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: This master's current role is: LEARNER
I20260927 12:35:47.681612 7341 catalog_manager.cc:1259] Loaded cluster ID: 21c7c51011ae44b1a1949b6e3297cf36
I20260927 12:35:47.681722 7341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: loading cluster ID for follower catalog manager: success
I20260927 12:35:47.683599 7341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: acquiring CA information for follower catalog manager: success
I20260927 12:35:47.685168 7341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:35:47.689273 6832 heartbeater.cc:503] Master 127.0.98.62:33875 was elected leader, sending a full tablet report...
W20260927 12:35:47.691928 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:47.692279 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:47.692348 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:47.698553 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:47.698652 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:47.698757 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:47.705416 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39351
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:47.706501 7251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:47.707479 7251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:47.717130 7349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.717183 7251 server_base.cc:1034] running on GCE node
W20260927 12:35:47.717134 7346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.717123 7347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.717780 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:47.718427 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:47.719580 7251 hybrid_clock.cc:648] HybridClock initialized: now 1790512547719558 us; error 33 us; skew 500 ppm
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:47.722450 7251 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:35:47.723459 7251 webserver.cc:460] Webserver started at http://127.0.98.1:39351/ using document root <none> and password file <none>
I20260927 12:35:47.724040 7251 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.724159 7251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.728019 7251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:47.730180 7356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.731495 7251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:35:47.731616 7251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "994b257ca30f4a6693158c2b3218bbf5"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "23a7c6ea2e37f8ba8a1a00095f79aefb"
server_key_iv: "bfce544d8f4748bb1aba90ae4100cf69"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.732232 7251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.744602 7251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:47.747943 7251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:47.748158 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:47.748755 7251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:47.749656 7251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:47.749706 7251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.749761 7251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:47.749788 7251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.759559 7251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35845
I20260927 12:35:47.759626 7469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35845 every 8 connection(s)
I20260927 12:35:47.760617 7251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:47.765848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 7251
I20260927 12:35:47.766057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6703
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:35:47.774709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35521
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33875
--webserver_port=40547
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:35:47.778873 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.764298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49723 (local address 127.0.98.61:43279)
0927 12:35:47.764428 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.764432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.764572 (+ 140us) server_negotiation.cc:407] Connection header received
0927 12:35:47.765056 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.765061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.765124 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.765179 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.767184 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.767905 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770079 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.770435 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.775421 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.775426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.775429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.775460 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.776972 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777558 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.777561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.777563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.777651 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777942 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.777945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.777947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.778152 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.778220 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.778603 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.778721 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:35:47.779275 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.763312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51295 (local address 127.0.98.62:33875)
0927 12:35:47.763450 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.763454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.763965 (+ 511us) server_negotiation.cc:407] Connection header received
0927 12:35:47.765025 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.765028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.765086 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.765158 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.766836 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.767625 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770081 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.770435 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.774385 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.774390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.774393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.774425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.776169 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.776828 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.776833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.776835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.776924 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777320 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.777324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.777325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.777569 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.777662 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.779004 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.779127 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:35:47.779757 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.763159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:51295)
0927 12:35:47.763871 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.763885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.764784 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.765340 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.765347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.766001 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.766673 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.766691 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.768323 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.768327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.769923 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.769936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.770711 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.771057 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.771831 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.771848 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.774138 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.776300 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.776308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.776329 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.776704 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.777019 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.777024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.777216 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.777676 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.777683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.778029 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.778237 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.778595 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":272,"spinlock_wait_cycles":35328,"thread_start_us":126,"threads_started":1}
I20260927 12:35:47.779809 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.764216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.98.1:49723)
0927 12:35:47.764465 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.764477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.764800 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.765350 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.765353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.766886 (+ 1533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.767022 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.767031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.768200 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.768205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.769935 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.769942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.770711 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.771082 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.772036 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.772052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.774549 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.777102 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.777107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.777443 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.777754 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.777758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.777853 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.778282 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.778285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.778415 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.779091 (+ 676us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.779223 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":299,"spinlock_wait_cycles":15872,"thread_start_us":80,"threads_started":1}
I20260927 12:35:47.780782 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.762598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35995 (local address 127.0.98.60:40475)
0927 12:35:47.762901 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.762904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.763938 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 12:35:47.765122 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.765126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.765182 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.765262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.767103 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.768174 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770072 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.770433 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770537 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.774615 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.774619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.774621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.774652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.776971 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777564 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.777567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.777569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.777651 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777942 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.777945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.777947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.778152 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.778234 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.780535 (+ 2301us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.780652 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260927 12:35:47.781121 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.763035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.98.1:35995)
0927 12:35:47.763630 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.763683 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.764805 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.765402 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.765405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.766870 (+ 1465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.766978 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.766985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.768298 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.768302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.769934 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.769945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.770709 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.770713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.771016 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.771689 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.771708 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.774424 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.777102 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.777107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.777443 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.777747 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.777750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.777752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.777843 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.778276 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.778279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.778414 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.778531 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.778681 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":335,"spinlock_wait_cycles":12544,"thread_start_us":642,"threads_started":1}
I20260927 12:35:47.781257 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33875
I20260927 12:35:47.781497 7470 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:47.781903 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40475
I20260927 12:35:47.781980 7471 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:47.782131 7471 heartbeater.cc:511] Master 127.0.98.60:40475 requested a full tablet report, sending...
I20260927 12:35:47.782158 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43279
I20260927 12:35:47.782230 7472 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:47.782359 7472 heartbeater.cc:511] Master 127.0.98.61:43279 requested a full tablet report, sending...
I20260927 12:35:47.782131 7470 heartbeater.cc:511] Master 127.0.98.62:33875 requested a full tablet report, sending...
I20260927 12:35:47.783833 6506 ts_manager.cc:194] Re-registered known tserver with Master: 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845)
I20260927 12:35:47.784019 7107 ts_manager.cc:194] Registered new tserver with Master: 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845)
I20260927 12:35:47.784816 7284 ts_manager.cc:194] Registered new tserver with Master: 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845)
I20260927 12:35:47.785787 6506 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51295
W20260927 12:35:47.896700 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:47.896981 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:47.897024 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:47.901136 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:47.901207 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:47.901286 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:47.906899 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40547
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:47.907958 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:47.909011 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:35:47.910210 6966 heartbeater.cc:503] Master 127.0.98.62:33875 was elected leader, sending a full tablet report...
W20260927 12:35:47.919209 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.919215 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:47.919224 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:47.921355 7478 server_base.cc:1034] running on GCE node
I20260927 12:35:47.921833 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:47.922516 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:47.923683 7478 hybrid_clock.cc:648] HybridClock initialized: now 1790512547923632 us; error 63 us; skew 500 ppm
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:47.926764 7478 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:35:47.927613 7478 webserver.cc:460] Webserver started at http://127.0.98.2:40547/ using document root <none> and password file <none>
I20260927 12:35:47.928213 7478 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:47.928292 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:47.932262 7478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:35:47.934340 7493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.935696 7478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:35:47.935801 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "44e154ece3954fdb81c20f6f6afbbd94"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "538af44cd7831a8f5ae7ab629bfc6925"
server_key_iv: "76d7d04a314531b60582fb369d083744"
server_key_version: "encryptionkey@0"
I20260927 12:35:47.936431 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:47.948308 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:47.951491 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:47.951669 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:47.952371 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:47.953258 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:47.953303 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.953344 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:47.953372 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:47.962584 7478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35521
I20260927 12:35:47.962606 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35521 every 8 connection(s)
I20260927 12:35:47.963630 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:47.972630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 7478
I20260927 12:35:47.972817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6837
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:35:47.980183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32803
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33875
--webserver_port=37015
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:35:47 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512547, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:47.981760 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.965581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37775 (local address 127.0.98.60:40475)
0927 12:35:47.965685 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.965689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.967294 (+ 1605us) server_negotiation.cc:407] Connection header received
0927 12:35:47.968770 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.968773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.968839 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.968939 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.970534 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.971351 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.972639 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.973001 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.973066 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.977311 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.977316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.977318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.977354 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.979327 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.979839 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.979843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.979845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.979950 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.980671 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.980675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.980677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.980886 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.980961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.981467 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.981599 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:35:47.981760 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.968338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50759 (local address 127.0.98.61:43279)
0927 12:35:47.968448 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.968453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.968871 (+ 418us) server_negotiation.cc:407] Connection header received
0927 12:35:47.968996 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.969000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.969048 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.969124 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.970515 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.971350 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.972526 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.972832 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.972910 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.977399 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.977403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.977405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.977433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.979022 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.979839 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.979843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.979845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.979950 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.980409 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.980412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.980414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.980647 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.980725 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.981467 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.981595 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:35:47.983680 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.968312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.98.2:50759)
0927 12:35:47.968694 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.968707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.968889 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.969291 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.969300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.969779 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.970357 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.970372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.971513 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.971516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.972365 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.972378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.973839 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.973842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.974054 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.974653 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.974677 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.977181 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.979183 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.979189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.979206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.979717 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.980056 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.980059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.980060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.980298 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.980787 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.980803 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.981184 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.981979 (+ 795us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.982387 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":82,"spinlock_wait_cycles":11776,"thread_start_us":208,"threads_started":1}
I20260927 12:35:47.984745 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.966212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.98.2:37775)
0927 12:35:47.967097 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.967146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.968364 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.969294 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.969300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.969815 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.970365 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.970375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.971464 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.971467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.972496 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.972506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.973839 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.973843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.974061 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.974803 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.974816 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.977102 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.979428 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.979431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.979433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.979727 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.980052 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.980055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.980057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.980565 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.981012 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.981015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.981183 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.982230 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.982455 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":255,"spinlock_wait_cycles":23680,"thread_start_us":693,"threads_started":1}
I20260927 12:35:47.985395 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43279
I20260927 12:35:47.985657 7609 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:47.985924 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40475
I20260927 12:35:47.986052 7608 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:47.986282 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.966212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:57527)
0927 12:35:47.967445 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:47.967458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:47.968361 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:47.969294 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:47.969300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:47.969814 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:47.970370 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.970375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.971216 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.971218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.972502 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:47.972508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.979804 (+ 7296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.979808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:47.979937 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.980446 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:47.980460 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:47.981946 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:47.983888 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.983891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.983893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.984133 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.984586 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:47.984589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:47.984591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:47.984860 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:47.985397 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:47.985411 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:47.985738 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 12:35:47.985949 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 12:35:47.986107 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":140,"spinlock_wait_cycles":131584,"thread_start_us":210,"threads_started":1}
I20260927 12:35:47.986320 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:47.965523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57527 (local address 127.0.98.62:33875)
0927 12:35:47.965659 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:47.965663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:47.967537 (+ 1874us) server_negotiation.cc:407] Connection header received
0927 12:35:47.968765 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:47.968770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:47.968839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:47.968908 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:47.970528 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.971065 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.972639 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:47.972980 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:47.973055 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:47.982127 (+ 9072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:47.982132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:47.982135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:47.982167 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:47.983740 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.984246 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.984250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.984252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.984358 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:47.984991 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:47.984995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:47.984997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:47.985247 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:47.985336 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:47.986039 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 12:35:47.986174 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:47.986295 7609 heartbeater.cc:511] Master 127.0.98.61:43279 requested a full tablet report, sending...
I20260927 12:35:47.986296 7608 heartbeater.cc:511] Master 127.0.98.60:40475 requested a full tablet report, sending...
I20260927 12:35:47.987111 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33875
I20260927 12:35:47.987187 7607 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:47.987345 7607 heartbeater.cc:511] Master 127.0.98.62:33875 requested a full tablet report, sending...
I20260927 12:35:47.987528 7107 ts_manager.cc:194] Registered new tserver with Master: 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521)
I20260927 12:35:47.987528 7284 ts_manager.cc:194] Registered new tserver with Master: 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521)
I20260927 12:35:47.987960 6506 ts_manager.cc:194] Re-registered known tserver with Master: 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521)
I20260927 12:35:47.988833 6506 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57527
I20260927 12:35:48.053360 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.049535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.0.98.60:40475)
0927 12:35:48.049657 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.049660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.049676 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:48.049878 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.049881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.049936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.050035 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:48.050355 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.050871 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.052401 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.052841 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.052935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.053063 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.053186 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.053247 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:48.053401 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.049425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38370)
0927 12:35:48.049579 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.049595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.049718 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.050073 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.050076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.050100 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:48.050215 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.050221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.050993 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.050997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.052255 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.052268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.052964 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.052967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.053066 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.053093 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.053171 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.053250 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 12:35:48.054121 7301 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 12:35:48.054525 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cbd7058d314252ada6b7fe971c641c. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:48.054666 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: This master's current role is: LEARNER
W20260927 12:35:48.101298 7614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:48.101603 7614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:48.101648 7614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:48.105852 7614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:48.105926 7614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:48.105988 7614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:48.111763 7614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37015
--tserver_master_addrs=127.0.98.62:33875,127.0.98.61:43279,127.0.98.60:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:48.112905 7614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:48.113899 7614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:48.123885 7619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:48.123929 7622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:48.123842 7620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:48.124231 7614 server_base.cc:1034] running on GCE node
I20260927 12:35:48.124696 7614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:48.125391 7614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:48.126582 7614 hybrid_clock.cc:648] HybridClock initialized: now 1790512548126554 us; error 40 us; skew 500 ppm
Sep 27 12:35:48 dist-test-slave-tzgx krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:48.129463 7614 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:35:48.130385 7614 webserver.cc:460] Webserver started at http://127.0.98.3:37015/ using document root <none> and password file <none>
I20260927 12:35:48.130934 7614 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:48.131014 7614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:48.135097 7614 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 12:35:48.137655 7629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:48.139082 7614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:48.139217 7614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "01de338cd64a4c45a45c8c21363c6183"
format_stamp: "Formatted at 2026-09-27 12:35:46 on dist-test-slave-tzgx"
server_key: "650ec2cfc2b38c434d34478f36572558"
server_key_iv: "3fd96a7c702654e13d7a59222d0f8c2a"
server_key_version: "encryptionkey@0"
I20260927 12:35:48.139851 7614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:48.151130 7614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:48.154376 7614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:48.154567 7614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:48.155200 7614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:48.156148 7614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:48.156200 7614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:48.156255 7614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:48.156282 7614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:48.166941 7614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32803
I20260927 12:35:48.166973 7742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32803 every 8 connection(s)
I20260927 12:35:48.168031 7614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:48.178184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 7614
Sep 27 12:35:48 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:35:48 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 12:35:48 dist-test-slave-tzgx krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512548, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:35:48.185160 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.169893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53013 (local address 127.0.98.62:33875)
0927 12:35:48.170016 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.170020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.171429 (+ 1409us) server_negotiation.cc:407] Connection header received
0927 12:35:48.172575 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.172578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.172631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.172681 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.174192 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.175010 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176447 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.176735 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176823 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.180842 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.180846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.180849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.180878 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.182714 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.183279 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.183282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.183284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.183401 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.183715 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.183718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.183720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.183951 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.184025 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.184781 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.184915 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:35:48.186308 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.170296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.98.3:46139)
0927 12:35:48.170917 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.170980 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.172308 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.172894 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.172903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.173436 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:48.174051 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.174058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.175117 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.175120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.176071 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.176081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.176597 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.176794 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.177694 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:48.177709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:48.180535 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:48.184219 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:48.184222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:48.184224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:48.184585 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:48.184912 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:48.184915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:48.184917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:48.185083 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:48.185534 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:48.185537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:48.185629 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.185722 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.185845 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":168,"spinlock_wait_cycles":16000,"thread_start_us":543,"threads_started":1}
I20260927 12:35:48.186600 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.171477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.98.3:47631)
0927 12:35:48.171790 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.171802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.172317 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.172894 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.172903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.173436 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:48.174047 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.174058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.174866 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.174868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.176135 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.176145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.176777 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.176853 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.177719 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:48.177731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:48.180230 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:48.182387 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:48.182395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:48.182417 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:48.182791 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:48.183097 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:48.183100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:48.183102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:48.183383 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:48.184336 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:48.184339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:48.184492 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.184660 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.185194 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":109,"spinlock_wait_cycles":4224,"thread_start_us":102,"threads_started":1}
I20260927 12:35:48.187170 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.171647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47631 (local address 127.0.98.61:43279)
0927 12:35:48.171772 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.171776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.171911 (+ 135us) server_negotiation.cc:407] Connection header received
0927 12:35:48.172661 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.172664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.172720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.172788 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.174204 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.174728 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176260 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.176559 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176629 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.180469 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.180474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.180477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.180511 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.182256 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.182908 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.182912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.182914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.183001 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.183536 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.183540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.183542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.183781 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.184237 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.186852 (+ 2615us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.187010 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:35:48.187549 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.170296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:53013)
0927 12:35:48.171317 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.171330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.172320 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.172894 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.172902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.173460 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:48.174041 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.174057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.175510 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.175527 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.176346 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.176354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.176855 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.176962 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.177592 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:48.177614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:48.180673 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:48.182855 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:48.182858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:48.182861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:48.183159 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:48.183524 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:48.183527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:48.183529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:48.183613 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:48.184132 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:48.184141 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:48.184447 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.184704 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.187385 (+ 2681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":2443,"spinlock_wait_cycles":23424,"thread_start_us":132,"threads_started":1}
I20260927 12:35:48.187650 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.169826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46139 (local address 127.0.98.60:40475)
0927 12:35:48.169958 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.169962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.171142 (+ 1180us) server_negotiation.cc:407] Connection header received
0927 12:35:48.172552 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.172556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.172613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.172692 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.174214 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.175004 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176212 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.176446 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.176550 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.181125 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.181131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.181134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.181165 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.183854 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.184723 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.184726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.184728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.184815 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.185219 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.185223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.185224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.185435 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.185542 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.187163 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.187425 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:35:48.188946 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33875
I20260927 12:35:48.189214 7743 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:48.189838 7743 heartbeater.cc:511] Master 127.0.98.62:33875 requested a full tablet report, sending...
I20260927 12:35:48.190412 7744 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40475
I20260927 12:35:48.190496 7744 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:48.190649 7744 heartbeater.cc:511] Master 127.0.98.60:40475 requested a full tablet report, sending...
I20260927 12:35:48.191849 6506 ts_manager.cc:194] Re-registered known tserver with Master: 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3:32803)
I20260927 12:35:48.192582 6506 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53013
I20260927 12:35:48.193336 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43279
I20260927 12:35:48.193434 7745 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:48.193637 7745 heartbeater.cc:511] Master 127.0.98.61:43279 requested a full tablet report, sending...
I20260927 12:35:48.194154 7284 ts_manager.cc:194] Registered new tserver with Master: 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3:32803)
I20260927 12:35:48.195838 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.180616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.0.98.62:33875)
0927 12:35:48.181611 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.181616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.185598 (+ 3982us) server_negotiation.cc:407] Connection header received
0927 12:35:48.185636 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.185639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.185685 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.185791 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.186926 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.187711 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.189121 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.189387 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.189628 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.191276 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.191279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.191282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.191311 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.193306 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.193824 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.193828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.193829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.194285 (+ 456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.194597 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.194600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.194602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.194825 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.194945 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.195497 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.195649 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":98,"threads_started":1}
I20260927 12:35:48.195861 7107 ts_manager.cc:194] Registered new tserver with Master: 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3:32803)
I20260927 12:35:48.197436 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.180788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.0.98.61:43279)
0927 12:35:48.183661 (+ 2873us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.183677 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.188078 (+ 4401us) server_negotiation.cc:407] Connection header received
0927 12:35:48.188197 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.188201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.188251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.188340 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.189473 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.190602 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.191785 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.192138 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.192232 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.193797 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.193801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.193804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.193830 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.195409 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.195867 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.195871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.195872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.196061 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.196367 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.196371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.196372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.196588 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.196707 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.197012 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.197277 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2799,"thread_start_us":137,"threads_started":1}
I20260927 12:35:48.198251 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.183022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.0.98.60:40475)
0927 12:35:48.185567 (+ 2545us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.185572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.189938 (+ 4366us) server_negotiation.cc:407] Connection header received
0927 12:35:48.190064 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.190067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.190113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.190185 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.191016 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.191851 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.193040 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.193395 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.193489 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.194962 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.194965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.194968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.194991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.196571 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.196935 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.196938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.196940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.197025 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.197309 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.197312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.197314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.197639 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.197769 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.197971 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.198099 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":89,"threads_started":1}
I20260927 12:35:48.201411 6506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:48.204447 6506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:48.207209 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: true } } }
I20260927 12:35:48.207369 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:48.210614 7067 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: attempting to promote NON_VOTER ad26acff36da44449acd6536d016bd9e to VOTER
I20260927 12:35:48.210918 7067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } }
I20260927 12:35:48.212698 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Refusing update from remote peer 79cbd7058d314252ada6b7fe971c641c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 12:35:48.213071 7762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 12:35:48.213685 7301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 LEARNER]: Refusing update from remote peer 79cbd7058d314252ada6b7fe971c641c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 12:35:48.214250 7762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 12:35:48.214486 7763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ad26acff36da44449acd6536d016bd9e (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.215862 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.215835 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.215993 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:48.216035 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:48.216843 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ad26acff36da44449acd6536d016bd9e (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.217839 7301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ad26acff36da44449acd6536d016bd9e (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.219089 7771 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.219216 7771 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:48.219599 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.219709 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:48.220125 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79cbd7058d314252ada6b7fe971c641c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:48.220240 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:48.227674 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.222234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32803 (local address 127.0.0.1:53192)
0927 12:35:48.222766 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.222780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.223168 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.223792 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.223796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.223821 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:48.223977 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.223983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.225253 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.225257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.226366 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.226384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.227203 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.227206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.227299 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.227320 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.227435 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.227514 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":77,"spinlock_wait_cycles":2304,"thread_start_us":106,"threads_started":1}
I20260927 12:35:48.227703 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.222941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.98.3:32803)
0927 12:35:48.223263 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.223274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.223301 (+ 27us) server_negotiation.cc:407] Connection header received
0927 12:35:48.223360 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.223365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.223503 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.223645 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:48.224199 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.224971 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.226492 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.227080 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.227185 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.227351 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.227477 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.227581 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":78,"threads_started":1}
I20260927 12:35:48.228615 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.221985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35845 (local address 127.0.0.1:51920)
0927 12:35:48.222200 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.222215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.222287 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.223800 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.223803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.223826 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:48.223966 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.223972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.225485 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.225489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.226375 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.226384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.228268 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.228272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.228367 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.228388 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.228434 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.228506 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":59,"spinlock_wait_cycles":3200}
I20260927 12:35:48.229113 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.222566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35521 (local address 127.0.0.1:34856)
0927 12:35:48.222806 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.222819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.222903 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.223628 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.223631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.223649 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:48.223796 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.223803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.225486 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.225489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.227123 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.227130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.228268 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.228272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.228820 (+ 548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.228842 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.228911 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.228989 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":1295,"thread_start_us":97,"threads_started":1}
I20260927 12:35:48.229190 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.222725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.0.98.2:35521)
0927 12:35:48.223117 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.223130 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.223154 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:35:48.223235 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.223241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.223479 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.223646 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:48.223923 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.225344 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.227244 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.228118 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.228319 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.228481 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.228955 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.229047 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":117,"threads_started":1}
I20260927 12:35:48.229758 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.222195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.0.98.1:35845)
0927 12:35:48.223387 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.223396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.223413 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:35:48.223474 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.223477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.223644 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.223834 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:48.224209 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.225342 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.227268 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.228144 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.229363 (+ 1219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.229498 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.229545 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.229630 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":130,"threads_started":1}
I20260927 12:35:48.231096 7677 tablet_service.cc:1467] Processing CreateTablet for tablet 81ea6df1d44540c7b0acb6eb45596b14 (DEFAULT_TABLE table=test-table [id=cb1645907e9942669a3e030c90020b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:48.232024 7677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ea6df1d44540c7b0acb6eb45596b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:48.231791 7541 tablet_service.cc:1467] Processing CreateTablet for tablet 81ea6df1d44540c7b0acb6eb45596b14 (DEFAULT_TABLE table=test-table [id=cb1645907e9942669a3e030c90020b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:48.232371 7404 tablet_service.cc:1467] Processing CreateTablet for tablet 81ea6df1d44540c7b0acb6eb45596b14 (DEFAULT_TABLE table=test-table [id=cb1645907e9942669a3e030c90020b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:48.232863 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ea6df1d44540c7b0acb6eb45596b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:48.233345 7404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ea6df1d44540c7b0acb6eb45596b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:48.239467 7777 tablet_bootstrap.cc:492] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Bootstrap starting.
I20260927 12:35:48.241966 7779 tablet_bootstrap.cc:492] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Bootstrap starting.
I20260927 12:35:48.242173 7777 tablet_bootstrap.cc:654] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:48.242894 7777 log.cc:824] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:48.243901 7779 tablet_bootstrap.cc:654] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:48.244468 7778 tablet_bootstrap.cc:492] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Bootstrap starting.
I20260927 12:35:48.244717 7779 log.cc:824] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:48.244987 7777 tablet_bootstrap.cc:492] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: No bootstrap required, opened a new log
I20260927 12:35:48.245191 7777 ts_tablet_manager.cc:1397] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:35:48.246608 7779 tablet_bootstrap.cc:492] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: No bootstrap required, opened a new log
I20260927 12:35:48.246760 7778 tablet_bootstrap.cc:654] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:48.246795 7779 ts_tablet_manager.cc:1397] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 12:35:48.247591 7778 log.cc:824] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:48.249851 7778 tablet_bootstrap.cc:492] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: No bootstrap required, opened a new log
I20260927 12:35:48.250056 7778 ts_tablet_manager.cc:1397] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:35:48.250329 7777 raft_consensus.cc:348] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.250615 7777 raft_consensus.cc:374] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:48.250674 7777 raft_consensus.cc:729] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01de338cd64a4c45a45c8c21363c6183, State: Initialized, Role: FOLLOWER
I20260927 12:35:48.250800 7779 raft_consensus.cc:348] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.251040 7779 raft_consensus.cc:374] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:48.251092 7779 raft_consensus.cc:729] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e154ece3954fdb81c20f6f6afbbd94, State: Initialized, Role: FOLLOWER
I20260927 12:35:48.251159 7777 consensus_queue.cc:260] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.251489 7779 consensus_queue.cc:260] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.251907 7743 heartbeater.cc:503] Master 127.0.98.62:33875 was elected leader, sending a full tablet report...
I20260927 12:35:48.252471 7779 ts_tablet_manager.cc:1428] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 12:35:48.252780 7607 heartbeater.cc:503] Master 127.0.98.62:33875 was elected leader, sending a full tablet report...
I20260927 12:35:48.255904 7777 ts_tablet_manager.cc:1428] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260927 12:35:48.256856 7778 raft_consensus.cc:348] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.257143 7778 raft_consensus.cc:374] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:48.257208 7778 raft_consensus.cc:729] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994b257ca30f4a6693158c2b3218bbf5, State: Initialized, Role: FOLLOWER
I20260927 12:35:48.257661 7778 consensus_queue.cc:260] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.258402 7470 heartbeater.cc:503] Master 127.0.98.62:33875 was elected leader, sending a full tablet report...
I20260927 12:35:48.260035 7778 ts_tablet_manager.cc:1428] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260927 12:35:48.262748 7473 tablet.cc:2369] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:48.318588 7783 raft_consensus.cc:482] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:48.318809 7783 raft_consensus.cc:504] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.319914 7783 leader_election.cc:288] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845), 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521)
I20260927 12:35:48.323700 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.320446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49605 (local address 127.0.98.1:35845)
0927 12:35:48.320558 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.320562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.320576 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:35:48.320630 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.320633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.320682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.320739 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:48.321053 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.321552 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.322651 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.323157 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.323251 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.323315 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.323516 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.323583 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:35:48.323700 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.320291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33451 (local address 127.0.98.2:35521)
0927 12:35:48.320477 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.320481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.320560 (+ 79us) server_negotiation.cc:407] Connection header received
0927 12:35:48.320682 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.320684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.320727 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.320799 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:48.321026 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.321543 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.322686 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.323111 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.323201 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.323267 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.323493 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.323574 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 12:35:48.323702 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.320285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35521 (local address 127.0.98.3:33451)
0927 12:35:48.320453 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.320467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.320589 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.320818 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.320821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.320838 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:48.320930 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.320934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.321635 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.321638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.322551 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.322561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.323240 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.323244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.323355 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.323379 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.323449 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.323526 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":39}
I20260927 12:35:48.323702 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.320190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35845 (local address 127.0.98.3:49605)
0927 12:35:48.320419 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:48.320434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:48.320530 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:48.320770 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:48.320773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:48.320787 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:48.320944 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.320949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.321656 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.321659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.322553 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:48.322563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.323325 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.323328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:48.323411 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.323428 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:35:48.323500 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:35:48.323553 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":58}
I20260927 12:35:48.324443 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea6df1d44540c7b0acb6eb45596b14" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e154ece3954fdb81c20f6f6afbbd94" is_pre_election: true
I20260927 12:35:48.324522 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea6df1d44540c7b0acb6eb45596b14" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994b257ca30f4a6693158c2b3218bbf5" is_pre_election: true
I20260927 12:35:48.324751 7561 raft_consensus.cc:2454] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 0.
I20260927 12:35:48.324875 7424 raft_consensus.cc:2454] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 0.
I20260927 12:35:48.325238 7630 leader_election.cc:302] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01de338cd64a4c45a45c8c21363c6183, 44e154ece3954fdb81c20f6f6afbbd94; no voters:
I20260927 12:35:48.325541 7783 raft_consensus.cc:2781] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:48.325618 7783 raft_consensus.cc:482] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:48.325664 7783 raft_consensus.cc:3037] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:48.326689 7783 raft_consensus.cc:504] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.327054 7783 leader_election.cc:288] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 election: Requested vote from peers 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845), 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521)
I20260927 12:35:48.327462 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea6df1d44540c7b0acb6eb45596b14" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e154ece3954fdb81c20f6f6afbbd94"
I20260927 12:35:48.327462 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ea6df1d44540c7b0acb6eb45596b14" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994b257ca30f4a6693158c2b3218bbf5"
I20260927 12:35:48.327605 7424 raft_consensus.cc:3037] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:48.327605 7561 raft_consensus.cc:3037] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:48.328909 7424 raft_consensus.cc:2454] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 1.
I20260927 12:35:48.328909 7561 raft_consensus.cc:2454] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 1.
I20260927 12:35:48.329239 7630 leader_election.cc:302] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01de338cd64a4c45a45c8c21363c6183, 44e154ece3954fdb81c20f6f6afbbd94, 994b257ca30f4a6693158c2b3218bbf5; no voters:
I20260927 12:35:48.329504 7783 raft_consensus.cc:2781] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:48.329694 7783 raft_consensus.cc:686] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 1 LEADER]: Becoming Leader. State: Replica: 01de338cd64a4c45a45c8c21363c6183, State: Running, Role: LEADER
I20260927 12:35:48.330076 7783 consensus_queue.cc:237] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } }
I20260927 12:35:48.333673 6506 catalog_manager.cc:5495] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "01de338cd64a4c45a45c8c21363c6183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:48.355614 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.351796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.98.3:32803)
0927 12:35:48.352119 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.352124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.352139 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:35:48.352212 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.352215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.352270 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.352367 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:48.352700 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.353235 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.354133 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.354362 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.354444 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.354694 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:48.355147 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:48.355236 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.355382 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.355443 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:35:48.362308 7424 raft_consensus.cc:1261] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Refusing update from remote peer 01de338cd64a4c45a45c8c21363c6183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:48.362308 7561 raft_consensus.cc:1261] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Refusing update from remote peer 01de338cd64a4c45a45c8c21363c6183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:48.363293 7783 consensus_queue.cc:1035] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:48.363531 7786 consensus_queue.cc:1035] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:48.376727 7791 mvcc.cc:205] Tried to move back new op lower bound from 7333939398083227648 to 7333939397962833920. Current Snapshot: MvccSnapshot[applied={T|T < 7333939398083227648}]
I20260927 12:35:48.384794 6505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:48.384984 6505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:48.389842 6505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79cbd7058d314252ada6b7fe971c641c: Sending DeleteTablet for 3 replicas of tablet 81ea6df1d44540c7b0acb6eb45596b14
I20260927 12:35:48.391004 7404 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ea6df1d44540c7b0acb6eb45596b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51920
I20260927 12:35:48.391001 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ea6df1d44540c7b0acb6eb45596b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34856
I20260927 12:35:48.391592 7800 tablet_replica.cc:330] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: stopping tablet replica
I20260927 12:35:48.391717 7677 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ea6df1d44540c7b0acb6eb45596b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53192
I20260927 12:35:48.391959 7800 raft_consensus.cc:2229] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:48.392251 7802 tablet_replica.cc:330] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: stopping tablet replica
I20260927 12:35:48.392354 7800 raft_consensus.cc:2258] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:48.392753 7801 tablet_replica.cc:330] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: stopping tablet replica
I20260927 12:35:48.393051 7801 raft_consensus.cc:2229] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:48.393376 7801 raft_consensus.cc:2258] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:48.394388 7800 ts_tablet_manager.cc:1905] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:48.394939 7801 ts_tablet_manager.cc:1905] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:48.395036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6475
I20260927 12:35:48.398403 7801 ts_tablet_manager.cc:1918] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:48.398399 7800 ts_tablet_manager.cc:1918] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:48.398512 7801 log.cc:1196] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/81ea6df1d44540c7b0acb6eb45596b14
I20260927 12:35:48.398653 7800 log.cc:1196] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/81ea6df1d44540c7b0acb6eb45596b14
I20260927 12:35:48.398844 7801 ts_tablet_manager.cc:1939] T 81ea6df1d44540c7b0acb6eb45596b14 P 994b257ca30f4a6693158c2b3218bbf5: Deleting consensus metadata
I20260927 12:35:48.399101 7800 ts_tablet_manager.cc:1939] T 81ea6df1d44540c7b0acb6eb45596b14 P 44e154ece3954fdb81c20f6f6afbbd94: Deleting consensus metadata
I20260927 12:35:48.400357 7802 raft_consensus.cc:2229] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:48.400736 7802 raft_consensus.cc:2258] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:35:48.401707 7496 connection.cc:535] server connection from 127.0.0.1:34856 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 12:35:48.402057 7357 connection.cc:535] server connection from 127.0.0.1:51920 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:35:48.402201 7802 ts_tablet_manager.cc:1905] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:48.405118 7802 ts_tablet_manager.cc:1918] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:48.405220 7802 log.cc:1196] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/81ea6df1d44540c7b0acb6eb45596b14
I20260927 12:35:48.405534 7802 ts_tablet_manager.cc:1939] T 81ea6df1d44540c7b0acb6eb45596b14 P 01de338cd64a4c45a45c8c21363c6183: Deleting consensus metadata
W20260927 12:35:48.406628 7633 connection.cc:441] server connection from 127.0.0.1:53192 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53192 (request call id 1) could send its response
I20260927 12:35:48.413599 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.394331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.0.98.61:43279)
0927 12:35:48.394460 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.394466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.403300 (+ 8834us) server_negotiation.cc:407] Connection header received
0927 12:35:48.403488 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.403491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.403547 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.403598 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.404663 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.405490 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.406680 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.407051 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.407125 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.409473 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.409477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.409479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.409508 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.411156 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.412231 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.412235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.412237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.412322 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.412599 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.412602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.412604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.412805 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.412914 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.413279 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.413428 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:35:48.416968 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.406189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.98.61:43279)
0927 12:35:48.406298 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.406301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.407096 (+ 795us) server_negotiation.cc:407] Connection header received
0927 12:35:48.407497 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.407499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.407560 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.407630 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.409010 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.409715 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.410975 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.411283 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.411363 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.412803 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.412806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.412808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.412831 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.414882 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.415504 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.415507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.415509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.415593 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.415913 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.415916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.415918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.416240 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.416351 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.416692 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.416807 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:35:48.416968 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.407358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.0.98.60:40475)
0927 12:35:48.407469 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.407473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.407485 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:35:48.407525 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.407528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.407572 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.407646 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.408592 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.409367 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.410492 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.410820 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.410906 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.412744 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.412749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.412752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.412779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.414932 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.415484 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.415488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.415490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.415593 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.415913 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.415916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.415918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.416295 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.416413 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.416692 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.416807 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:35:48.422853 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:48.394451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.0.98.60:40475)
0927 12:35:48.394560 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:48.394565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:48.413623 (+ 19058us) server_negotiation.cc:407] Connection header received
0927 12:35:48.414615 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:48.414619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:48.414673 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:48.414721 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:48.415550 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.416388 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.417616 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:48.417958 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:48.418037 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:48.419815 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:48.419818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:48.419820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:48.419842 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:48.421157 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.421573 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.421576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.421577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.421644 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:48.421887 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:48.421890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:48.421891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:48.422106 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:48.422225 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:48.422564 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 12:35:48.422697 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:35:49.377151 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.376617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:43723)
0927 12:35:49.376931 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.377041 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":77,"threads_started":1}
W20260927 12:35:49.377543 7470 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
I20260927 12:35:49.377725 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.377262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:43529)
0927 12:35:49.377534 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.377632 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
W20260927 12:35:49.378052 7607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
I20260927 12:35:49.381274 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.380848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:53395)
0927 12:35:49.381073 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.381163 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
W20260927 12:35:49.382017 7743 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
I20260927 12:35:49.890120 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79cbd7058d314252ada6b7fe971c641c)
I20260927 12:35:49.890353 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } }
I20260927 12:35:49.890688 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79cbd7058d314252ada6b7fe971c641c)
I20260927 12:35:49.890844 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } }
I20260927 12:35:49.891458 7814 leader_election.cc:288] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875), c77fa177dd194f87bb762f1777a84b9a (127.0.98.61:43279)
I20260927 12:35:49.892067 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.891664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47244)
0927 12:35:49.891894 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.891968 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260927 12:35:49.892460 7815 leader_election.cc:288] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875), ad26acff36da44449acd6536d016bd9e (127.0.98.60:40475)
I20260927 12:35:49.892568 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.891771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47260)
0927 12:35:49.892372 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.892440 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":58,"threads_started":1}
W20260927 12:35:49.892714 7278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
W20260927 12:35:49.892748 7094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
I20260927 12:35:49.893544 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.893336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47264)
0927 12:35:49.893430 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.893479 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 12:35:49.893622 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.893388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47270)
0927 12:35:49.893522 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.893566 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 12:35:49.893671 7094 leader_election.cc:334] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
W20260927 12:35:49.893755 7278 leader_election.cc:334] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
I20260927 12:35:49.895738 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.891708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43279 (local address 127.0.0.1:56986)
0927 12:35:49.892029 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.892044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:49.892180 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:49.892584 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:49.892588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:49.892607 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:49.892719 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.892727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.893769 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.893772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.894643 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.894653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.895419 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.895421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.895497 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.895530 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 12:35:49.895586 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:35:49.895642 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260927 12:35:49.895784 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.891948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.0.98.61:43279)
0927 12:35:49.892329 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:49.892334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:49.892350 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:49.892399 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:49.892402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:49.892448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:49.892828 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:49.892922 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.893645 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.894744 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.895314 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.895413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.895472 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:49.895608 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 12:35:49.895676 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":47,"threads_started":1}
I20260927 12:35:49.896203 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad26acff36da44449acd6536d016bd9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c77fa177dd194f87bb762f1777a84b9a" is_pre_election: true
I20260927 12:35:49.896389 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad26acff36da44449acd6536d016bd9e in term 1.
I20260927 12:35:49.896693 7275 leader_election.cc:302] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad26acff36da44449acd6536d016bd9e, c77fa177dd194f87bb762f1777a84b9a; no voters: 79cbd7058d314252ada6b7fe971c641c
I20260927 12:35:49.896891 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 12:35:49.896907 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.892321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40475 (local address 127.0.0.1:38416)
0927 12:35:49.892689 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.892701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:49.892778 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:49.893207 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:49.893210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:49.893227 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:49.893313 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.893320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.894566 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.894569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.895711 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.895720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.896539 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.896543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.896637 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.896657 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:35:49.896721 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:35:49.896790 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":53,"threads_started":1}
I20260927 12:35:49.896909 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.892445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.0.98.60:40475)
0927 12:35:49.892997 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:49.893002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:49.893014 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:35:49.893053 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:49.893055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:49.893095 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:49.893752 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:49.893841 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.894460 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.895939 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.896425 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.896537 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.896590 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:49.896741 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 12:35:49.896806 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":479,"thread_start_us":74,"threads_started":1}
I20260927 12:35:49.896939 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 79cbd7058d314252ada6b7fe971c641c)
I20260927 12:35:49.896976 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:35:49.897375 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c77fa177dd194f87bb762f1777a84b9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ad26acff36da44449acd6536d016bd9e" is_pre_election: true
I20260927 12:35:49.898044 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } }
I20260927 12:35:49.898262 7301 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c77fa177dd194f87bb762f1777a84b9a in current term 2: Already voted for candidate ad26acff36da44449acd6536d016bd9e in this term.
I20260927 12:35:49.898398 7814 leader_election.cc:288] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [CANDIDATE]: Term 2 election: Requested vote from peers 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875), c77fa177dd194f87bb762f1777a84b9a (127.0.98.61:43279)
I20260927 12:35:49.898661 7091 leader_election.cc:302] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c77fa177dd194f87bb762f1777a84b9a; no voters: 79cbd7058d314252ada6b7fe971c641c, ad26acff36da44449acd6536d016bd9e
I20260927 12:35:49.898774 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.898526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47274)
0927 12:35:49.898665 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.898715 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 12:35:49.898739 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad26acff36da44449acd6536d016bd9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c77fa177dd194f87bb762f1777a84b9a"
I20260927 12:35:49.898849 7109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:35:49.899415 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.899217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47278)
0927 12:35:49.899312 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.899360 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 12:35:49.899554 7278 leader_election.cc:334] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
I20260927 12:35:49.899964 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad26acff36da44449acd6536d016bd9e in term 2.
I20260927 12:35:49.900085 7815 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:35:49.900302 7275 leader_election.cc:302] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad26acff36da44449acd6536d016bd9e, c77fa177dd194f87bb762f1777a84b9a; no voters: 79cbd7058d314252ada6b7fe971c641c
I20260927 12:35:49.900470 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:35:49.900691 7814 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [term 2 LEADER]: Becoming Leader. State: Replica: ad26acff36da44449acd6536d016bd9e, State: Running, Role: LEADER
I20260927 12:35:49.900934 7814 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } }
I20260927 12:35:49.902158 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad26acff36da44449acd6536d016bd9e. Latest consensus state: current_term: 2 leader_uuid: "ad26acff36da44449acd6536d016bd9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:49.902259 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:49.902524 7826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:49.904970 7826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:49.905359 7826 catalog_manager.cc:1259] Loaded cluster ID: 21c7c51011ae44b1a1949b6e3297cf36
I20260927 12:35:49.905402 7826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:49.905900 7826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:49.906251 7826 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Loaded TSK: 0
I20260927 12:35:49.906893 7826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:35:49.927875 7291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:49.930932 7291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:49.933898 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.933650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47304)
0927 12:35:49.933757 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.933817 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 12:35:49.934006 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [term 2 FOLLOWER]: Refusing update from remote peer ad26acff36da44449acd6536d016bd9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 12:35:49.934514 7814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 12:35:49.934593 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.934391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.0.1:47318)
0927 12:35:49.934493 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.934538 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 12:35:49.934736 7278 consensus_peers.cc:487] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e -> Peer 79cbd7058d314252ada6b7fe971c641c (127.0.98.62:33875): Couldn't send request to peer 79cbd7058d314252ada6b7fe971c641c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:35:49.936013 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad26acff36da44449acd6536d016bd9e. Latest consensus state: current_term: 2 leader_uuid: "ad26acff36da44449acd6536d016bd9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:49.936183 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:49.937232 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ad26acff36da44449acd6536d016bd9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:49.937361 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:49.937508 7828 mvcc.cc:205] Tried to move back new op lower bound from 7333939404525006848 to 7333939404396392448. Current Snapshot: MvccSnapshot[applied={T|T < 7333939404525006848}]
I20260927 12:35:49.937819 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ad26acff36da44449acd6536d016bd9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "79cbd7058d314252ada6b7fe971c641c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33875 } } peers { permanent_uuid: "c77fa177dd194f87bb762f1777a84b9a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43279 } attrs { promote: false } } peers { permanent_uuid: "ad26acff36da44449acd6536d016bd9e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40475 } attrs { promote: false } } }
I20260927 12:35:49.937937 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77fa177dd194f87bb762f1777a84b9a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:35:49.948676 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.943905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.0.98.2:35521)
0927 12:35:49.944688 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:49.944693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:49.944705 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:35:49.944748 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:49.944750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:49.944790 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:49.944978 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:49.945110 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.945879 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.947463 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.948123 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.948233 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.948301 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:49.948456 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 12:35:49.948542 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":77,"threads_started":1}
I20260927 12:35:49.948673 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.943528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35521 (local address 127.0.0.1:34870)
0927 12:35:49.943650 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.943665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:49.943753 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:49.944891 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:49.944894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:49.944917 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:49.945024 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.945031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.945995 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.945998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.947302 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.947312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.948230 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.948234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.948322 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.948350 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 12:35:49.948443 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 12:35:49.948525 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":69,"spinlock_wait_cycles":3840}
I20260927 12:35:49.948858 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.944081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32803 (local address 127.0.0.1:53202)
0927 12:35:49.944435 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.944446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:49.944515 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:49.944890 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:49.944893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:49.944926 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:49.945038 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.945043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.946693 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.946697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.947717 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.947725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.948486 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.948489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.948564 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.948583 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:35:49.948626 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 12:35:49.948680 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":22,"spinlock_wait_cycles":1280,"thread_start_us":101,"threads_started":1}
I20260927 12:35:49.948714 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.943786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35845 (local address 127.0.0.1:51934)
0927 12:35:49.944344 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.944357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:49.944446 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:49.944819 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:49.944822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:49.944838 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:49.944958 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.944965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.945995 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.945998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.947302 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:49.947312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.948359 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.948363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:49.948448 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.948471 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:35:49.948534 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:35:49.948601 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":77}
I20260927 12:35:49.949262 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.944445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.0.98.3:32803)
0927 12:35:49.944682 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:49.944686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:49.944699 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:49.944747 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:49.944750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:49.944794 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:49.944873 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:49.946019 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.946599 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.947825 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.948362 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.948963 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.949030 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:49.949063 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 12:35:49.949135 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260927 12:35:49.949388 7541 tablet_service.cc:1467] Processing CreateTablet for tablet a4f4a615bf3b4d60a8a71fe944bb15b5 (DEFAULT_TABLE table=test-table [id=2761ada1aa9b4944a56068ef0ec46921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:49.949537 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.944025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.0.98.1:35845)
0927 12:35:49.944324 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:49.944330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:49.944529 (+ 199us) server_negotiation.cc:407] Connection header received
0927 12:35:49.944588 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:49.944590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:49.944638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:49.944708 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:49.945085 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.945879 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.947463 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:49.948151 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:49.948247 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:49.948301 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:49.949324 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 12:35:49.949412 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":132,"threads_started":1}
I20260927 12:35:49.949702 7541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f4a615bf3b4d60a8a71fe944bb15b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:49.949707 7677 tablet_service.cc:1467] Processing CreateTablet for tablet a4f4a615bf3b4d60a8a71fe944bb15b5 (DEFAULT_TABLE table=test-table [id=2761ada1aa9b4944a56068ef0ec46921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:49.949941 7677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f4a615bf3b4d60a8a71fe944bb15b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:49.950009 7404 tablet_service.cc:1467] Processing CreateTablet for tablet a4f4a615bf3b4d60a8a71fe944bb15b5 (DEFAULT_TABLE table=test-table [id=2761ada1aa9b4944a56068ef0ec46921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:49.950237 7404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f4a615bf3b4d60a8a71fe944bb15b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:49.951613 7836 tablet_bootstrap.cc:492] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Bootstrap starting.
I20260927 12:35:49.952508 7838 tablet_bootstrap.cc:492] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: Bootstrap starting.
I20260927 12:35:49.952517 7836 tablet_bootstrap.cc:654] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:49.953210 7838 tablet_bootstrap.cc:654] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:49.953852 7837 tablet_bootstrap.cc:492] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Bootstrap starting.
I20260927 12:35:49.953871 7836 tablet_bootstrap.cc:492] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: No bootstrap required, opened a new log
I20260927 12:35:49.953958 7836 ts_tablet_manager.cc:1397] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:49.954368 7836 raft_consensus.cc:348] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:49.954488 7836 raft_consensus.cc:374] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:49.954528 7836 raft_consensus.cc:729] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e154ece3954fdb81c20f6f6afbbd94, State: Initialized, Role: FOLLOWER
I20260927 12:35:49.954684 7837 tablet_bootstrap.cc:654] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:49.954659 7836 consensus_queue.cc:260] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:49.955338 7836 ts_tablet_manager.cc:1428] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 12:35:49.955720 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.955284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:55027)
0927 12:35:49.955562 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.955621 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260927 12:35:49.956406 7838 tablet_bootstrap.cc:492] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: No bootstrap required, opened a new log
I20260927 12:35:49.956496 7838 ts_tablet_manager.cc:1397] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:35:49.956948 7838 raft_consensus.cc:348] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:49.957047 7838 raft_consensus.cc:374] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:49.957079 7838 raft_consensus.cc:729] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994b257ca30f4a6693158c2b3218bbf5, State: Initialized, Role: FOLLOWER
I20260927 12:35:49.957167 7608 heartbeater.cc:503] Master 127.0.98.60:40475 was elected leader, sending a full tablet report...
I20260927 12:35:49.957224 7838 consensus_queue.cc:260] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:49.957556 7838 ts_tablet_manager.cc:1428] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:49.957583 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.956175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:50545)
0927 12:35:49.957342 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.957514 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1103}
I20260927 12:35:49.958345 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.957830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:38991)
0927 12:35:49.958186 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.958239 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":93,"threads_started":1}
I20260927 12:35:49.959147 7471 heartbeater.cc:503] Master 127.0.98.60:40475 was elected leader, sending a full tablet report...
I20260927 12:35:49.959182 7837 tablet_bootstrap.cc:492] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: No bootstrap required, opened a new log
I20260927 12:35:49.959259 7837 ts_tablet_manager.cc:1397] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 12:35:49.959681 7837 raft_consensus.cc:348] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:49.959820 7837 raft_consensus.cc:374] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:49.959854 7837 raft_consensus.cc:729] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01de338cd64a4c45a45c8c21363c6183, State: Initialized, Role: FOLLOWER
I20260927 12:35:49.959978 7837 consensus_queue.cc:260] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:49.960294 7837 ts_tablet_manager.cc:1428] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:49.960552 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.958728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:47591)
0927 12:35:49.960431 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.960491 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1637}
I20260927 12:35:49.961052 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:49.960678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:40065)
0927 12:35:49.960894 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:49.960948 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260927 12:35:49.961712 7744 heartbeater.cc:503] Master 127.0.98.60:40475 was elected leader, sending a full tablet report...
W20260927 12:35:49.967108 7610 tablet.cc:2369] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:35:50.171090 7746 tablet.cc:2369] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:50.244120 7846 raft_consensus.cc:482] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:50.244269 7846 raft_consensus.cc:504] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:50.244738 7846 leader_election.cc:288] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521), 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845)
I20260927 12:35:50.245280 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f4a615bf3b4d60a8a71fe944bb15b5" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e154ece3954fdb81c20f6f6afbbd94" is_pre_election: true
I20260927 12:35:50.245280 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f4a615bf3b4d60a8a71fe944bb15b5" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994b257ca30f4a6693158c2b3218bbf5" is_pre_election: true
I20260927 12:35:50.245446 7561 raft_consensus.cc:2454] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 0.
I20260927 12:35:50.245446 7424 raft_consensus.cc:2454] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 0.
I20260927 12:35:50.245864 7630 leader_election.cc:302] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01de338cd64a4c45a45c8c21363c6183, 44e154ece3954fdb81c20f6f6afbbd94, 994b257ca30f4a6693158c2b3218bbf5; no voters:
I20260927 12:35:50.246071 7846 raft_consensus.cc:2781] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:50.246140 7846 raft_consensus.cc:482] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:50.246179 7846 raft_consensus.cc:3037] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:50.247275 7846 raft_consensus.cc:504] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:50.247649 7846 leader_election.cc:288] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 election: Requested vote from peers 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521), 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845)
I20260927 12:35:50.248136 7561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f4a615bf3b4d60a8a71fe944bb15b5" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e154ece3954fdb81c20f6f6afbbd94"
I20260927 12:35:50.248194 7424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f4a615bf3b4d60a8a71fe944bb15b5" candidate_uuid: "01de338cd64a4c45a45c8c21363c6183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "994b257ca30f4a6693158c2b3218bbf5"
I20260927 12:35:50.248306 7424 raft_consensus.cc:3037] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:50.248291 7561 raft_consensus.cc:3037] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:50.249531 7561 raft_consensus.cc:2454] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 1.
I20260927 12:35:50.249531 7424 raft_consensus.cc:2454] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01de338cd64a4c45a45c8c21363c6183 in term 1.
I20260927 12:35:50.249960 7630 leader_election.cc:302] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01de338cd64a4c45a45c8c21363c6183, 44e154ece3954fdb81c20f6f6afbbd94, 994b257ca30f4a6693158c2b3218bbf5; no voters:
I20260927 12:35:50.250304 7846 raft_consensus.cc:2781] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:50.250586 7846 raft_consensus.cc:686] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 1 LEADER]: Becoming Leader. State: Replica: 01de338cd64a4c45a45c8c21363c6183, State: Running, Role: LEADER
I20260927 12:35:50.250751 7846 consensus_queue.cc:237] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } }
I20260927 12:35:50.251540 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.251220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:35915)
0927 12:35:50.251377 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.251451 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:35:50.252753 7291 catalog_manager.cc:5495] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "01de338cd64a4c45a45c8c21363c6183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01de338cd64a4c45a45c8c21363c6183" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32803 } health_report { overall_health: HEALTHY } } }
I20260927 12:35:50.316032 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.312769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.0.98.3:32803)
0927 12:35:50.312890 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:50.312894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:50.312927 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:35:50.312973 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:50.312975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:50.313022 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:50.313086 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:50.313398 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:50.313914 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:50.314853 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:50.315102 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:50.315187 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:50.315499 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:50.315585 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:50.315636 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:50.315857 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:35:50.315916 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:35:50.318333 7424 raft_consensus.cc:1261] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Refusing update from remote peer 01de338cd64a4c45a45c8c21363c6183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:50.318356 7561 raft_consensus.cc:1261] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Refusing update from remote peer 01de338cd64a4c45a45c8c21363c6183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:50.318790 7848 consensus_queue.cc:1035] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "994b257ca30f4a6693158c2b3218bbf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:50.319020 7846 consensus_queue.cc:1035] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e154ece3954fdb81c20f6f6afbbd94" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:50.321336 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.320660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:35599)
0927 12:35:50.321180 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.321250 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447}
I20260927 12:35:50.322073 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.321127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:33417)
0927 12:35:50.321928 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.321996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":733}
I20260927 12:35:50.322974 7851 mvcc.cc:205] Tried to move back new op lower bound from 7333939406099443712 to 7333939405829210112. Current Snapshot: MvccSnapshot[applied={T|T < 7333939406099443712}]
I20260927 12:35:50.324036 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.323781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.2:42385)
0927 12:35:50.323923 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.323970 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 12:35:50.324121 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.322509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:59531)
0927 12:35:50.323975 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.324031 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1384}
I20260927 12:35:50.326792 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.326541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.3:55481)
0927 12:35:50.326672 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.326722 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 12:35:50.327616 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.327357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33875 (local address 127.0.98.1:33165)
0927 12:35:50.327495 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.327545 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 12:35:50.329221 7291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:50.329280 7852 mvcc.cc:205] Tried to move back new op lower bound from 7333939406099443712 to 7333939405829210112. Current Snapshot: MvccSnapshot[applied={T|T < 7333939406099443712}]
I20260927 12:35:50.329380 7291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:50.333067 7291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad26acff36da44449acd6536d016bd9e: Sending DeleteTablet for 3 replicas of tablet a4f4a615bf3b4d60a8a71fe944bb15b5
I20260927 12:35:50.333796 7541 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f4a615bf3b4d60a8a71fe944bb15b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:50 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34870
I20260927 12:35:50.333918 7677 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f4a615bf3b4d60a8a71fe944bb15b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:50 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53202
I20260927 12:35:50.333979 7404 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f4a615bf3b4d60a8a71fe944bb15b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:50 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51934
I20260927 12:35:50.334208 7861 tablet_replica.cc:330] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: stopping tablet replica
I20260927 12:35:50.334290 7863 tablet_replica.cc:330] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: stopping tablet replica
I20260927 12:35:50.334349 7861 raft_consensus.cc:2229] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:50.334396 7863 raft_consensus.cc:2229] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:50.334488 7861 raft_consensus.cc:2258] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:50.334522 7863 raft_consensus.cc:2258] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:50.334945 7862 tablet_replica.cc:330] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: stopping tablet replica
I20260927 12:35:50.335070 7862 raft_consensus.cc:2229] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:50.335101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 7251
I20260927 12:35:50.335162 7861 ts_tablet_manager.cc:1905] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:50.335203 7863 ts_tablet_manager.cc:1905] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 994b257ca30f4a6693158c2b3218bbf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:50.336302 7862 raft_consensus.cc:2258] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:50.337057 7862 ts_tablet_manager.cc:1905] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:50.337733 7861 ts_tablet_manager.cc:1918] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:50.337806 7861 log.cc:1196] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/a4f4a615bf3b4d60a8a71fe944bb15b5
I20260927 12:35:50.338037 7861 ts_tablet_manager.cc:1939] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 44e154ece3954fdb81c20f6f6afbbd94: Deleting consensus metadata
I20260927 12:35:50.338727 7275 catalog_manager.cc:4841] TS 44e154ece3954fdb81c20f6f6afbbd94 (127.0.98.2:35521): tablet a4f4a615bf3b4d60a8a71fe944bb15b5 (table test-table [id=2761ada1aa9b4944a56068ef0ec46921]) successfully deleted
I20260927 12:35:50.339718 7862 ts_tablet_manager.cc:1918] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:50.339797 7862 log.cc:1196] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/a4f4a615bf3b4d60a8a71fe944bb15b5
I20260927 12:35:50.340063 7862 ts_tablet_manager.cc:1939] T a4f4a615bf3b4d60a8a71fe944bb15b5 P 01de338cd64a4c45a45c8c21363c6183: Deleting consensus metadata
I20260927 12:35:50.340752 7275 catalog_manager.cc:4841] TS 01de338cd64a4c45a45c8c21363c6183 (127.0.98.3:32803): tablet a4f4a615bf3b4d60a8a71fe944bb15b5 (table test-table [id=2761ada1aa9b4944a56068ef0ec46921]) successfully deleted
I20260927 12:35:50.342460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 7478
I20260927 12:35:50.343472 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:50.343158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35845 (local address 127.0.0.1:51936)
0927 12:35:50.343299 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:50.343366 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 12:35:50.343659 7278 catalog_manager.cc:4584] TS 994b257ca30f4a6693158c2b3218bbf5 (127.0.98.1:35845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4f4a615bf3b4d60a8a71fe944bb15b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35845: connect: Connection refused (error 111)
I20260927 12:35:50.349509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 7614
I20260927 12:35:50.356457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 6971
I20260927 12:35:50.362021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 7074
2026-09-27T12:35:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5468 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:50 dist-test-slave-tzgx krb5kdc[7866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:50 dist-test-slave-tzgx krb5kdc[7866](info): set up 2 sockets
Sep 27 12:35:50 dist-test-slave-tzgx krb5kdc[7866](info): commencing operation
krb5kdc: starting...
W20260927 12:35:51.340629 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:51 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 12:35:51.467231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:51.581791 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:51.582120 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:51.582185 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:51.586483 7879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:51.586567 7879 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:35:51.586591 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:51.586614 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:51.586637 7879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:51.586666 7879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:51.592458 7879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39247
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:51.593606 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:51.594666 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:51.603541 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:51.603557 7879 server_base.cc:1034] running on GCE node
W20260927 12:35:51.603543 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:51.603559 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:51.604501 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:51.605651 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:51.606899 7879 hybrid_clock.cc:648] HybridClock initialized: now 1790512551606878 us; error 35 us; skew 500 ppm
Sep 27 12:35:51 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:51.616060 7879 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 12:35:51.617149 7879 webserver.cc:460] Webserver started at http://127.0.98.62:41819/ using document root <none> and password file <none>
I20260927 12:35:51.617722 7879 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:51.617787 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:51.617985 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:51.620054 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "960b1329d24c444c9ca6d8f8041ea2cb"
format_stamp: "Formatted at 2026-09-27 12:35:51 on dist-test-slave-tzgx"
server_key: "2cfaddcbe92564f9530736683a9d95f6"
server_key_iv: "3077d64dfb43a84c443ddc1cb871dfb3"
server_key_version: "encryptionkey@0"
I20260927 12:35:51.620666 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "960b1329d24c444c9ca6d8f8041ea2cb"
format_stamp: "Formatted at 2026-09-27 12:35:51 on dist-test-slave-tzgx"
server_key: "2cfaddcbe92564f9530736683a9d95f6"
server_key_iv: "3077d64dfb43a84c443ddc1cb871dfb3"
server_key_version: "encryptionkey@0"
I20260927 12:35:51.624482 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:35:51.627199 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:51.628643 7879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:51.628768 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "960b1329d24c444c9ca6d8f8041ea2cb"
format_stamp: "Formatted at 2026-09-27 12:35:51 on dist-test-slave-tzgx"
server_key: "2cfaddcbe92564f9530736683a9d95f6"
server_key_iv: "3077d64dfb43a84c443ddc1cb871dfb3"
server_key_version: "encryptionkey@0"
I20260927 12:35:51.628873 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:51.647715 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:51.651254 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:51.651463 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:51.659477 7879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39247
I20260927 12:35:51.659520 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39247 every 8 connection(s)
I20260927 12:35:51.660549 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:51.663707 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:51.664270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 7879
I20260927 12:35:51.664395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:51.664675 392 external_mini_cluster.cc:1428] Setting key 06d0f7e1c30f4ed3792d1c4210b7bfdc
I20260927 12:35:51.671509 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Bootstrap starting.
I20260927 12:35:51.673965 7957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Neither blocks nor log segments found. Creating new log.
Sep 27 12:35:51 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:51.674729 7957 log.cc:824] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:51.676786 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: No bootstrap required, opened a new log
I20260927 12:35:51.680001 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39247 } }
I20260927 12:35:51.680346 7957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:51.680403 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960b1329d24c444c9ca6d8f8041ea2cb, State: Initialized, Role: FOLLOWER
I20260927 12:35:51.680634 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:51.666224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.0.98.62:39247)
0927 12:35:51.666710 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:51.666731 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:51.666778 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:35:51.667525 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:51.667552 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:51.668021 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:51.668435 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:51.669332 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.670838 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.671773 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.672356 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.672510 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:51.675067 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:51.675073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:51.675088 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:51.675120 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:51.677494 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:51.677992 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:51.677999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:51.678006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:51.678120 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:51.678388 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:51.678392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:51.678393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:51.678884 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:51.679073 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:51.679396 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 12:35:51.679694 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":147,"threads_started":1}
I20260927 12:35:51.680923 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39247 } }
I20260927 12:35:51.681063 7957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:51.681125 7957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:51.681226 7957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:51.682193 7957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39247 } }
I20260927 12:35:51.682543 7957 leader_election.cc:302] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 960b1329d24c444c9ca6d8f8041ea2cb; no voters:
I20260927 12:35:51.682912 7957 leader_election.cc:288] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:51.683014 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:51.683257 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [term 1 LEADER]: Becoming Leader. State: Replica: 960b1329d24c444c9ca6d8f8041ea2cb, State: Running, Role: LEADER
I20260927 12:35:51.683563 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39247 } }
I20260927 12:35:51.684159 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:51.685353 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39247 } } }
I20260927 12:35:51.685499 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:51.685356 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 960b1329d24c444c9ca6d8f8041ea2cb. Latest consensus state: current_term: 1 leader_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "960b1329d24c444c9ca6d8f8041ea2cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39247 } } }
I20260927 12:35:51.685707 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:51.685950 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:51.688879 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:51.695206 7971 catalog_manager.cc:1347] Generated new cluster ID: af4c6f79d625424ca2a67c57e5d89b31
I20260927 12:35:51.695290 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:51.707574 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:51.708913 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:51.722420 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Generated new TSK 0
I20260927 12:35:51.723227 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 12:35:51.760640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39247
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:51.890133 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:51.890426 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:51.890471 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:51.894848 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:51.894948 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:51.895001 7983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:51.895043 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:51.900666 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:51.901736 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:51.902727 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:51.912537 7989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:51.912537 7988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:51.912539 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:51.912940 7983 server_base.cc:1034] running on GCE node
I20260927 12:35:51.913352 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:51.914012 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:51.915230 7983 hybrid_clock.cc:648] HybridClock initialized: now 1790512551915200 us; error 36 us; skew 500 ppm
Sep 27 12:35:51 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:51.918869 7983 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:35:51.920027 7983 webserver.cc:460] Webserver started at http://127.0.98.1:45801/ using document root <none> and password file <none>
I20260927 12:35:51.920759 7983 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:51.920854 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:51.921101 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:51.923889 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c57a90c244645e9a6ad7d1533fd983c"
format_stamp: "Formatted at 2026-09-27 12:35:51 on dist-test-slave-tzgx"
server_key: "7f6dcae8a15b144666d5e994f54fc017"
server_key_iv: "1928fb66c45ad85e26b1987a610ea946"
server_key_version: "encryptionkey@0"
I20260927 12:35:51.924592 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c57a90c244645e9a6ad7d1533fd983c"
format_stamp: "Formatted at 2026-09-27 12:35:51 on dist-test-slave-tzgx"
server_key: "7f6dcae8a15b144666d5e994f54fc017"
server_key_iv: "1928fb66c45ad85e26b1987a610ea946"
server_key_version: "encryptionkey@0"
I20260927 12:35:51.928737 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:35:51.931412 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:51.932891 7983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:51.933025 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "6c57a90c244645e9a6ad7d1533fd983c"
format_stamp: "Formatted at 2026-09-27 12:35:51 on dist-test-slave-tzgx"
server_key: "7f6dcae8a15b144666d5e994f54fc017"
server_key_iv: "1928fb66c45ad85e26b1987a610ea946"
server_key_version: "encryptionkey@0"
I20260927 12:35:51.933166 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:51.951280 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:51.954607 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:51.954805 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:51.957551 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:51.957623 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:51.957666 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:51.957680 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:51 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:51.968374 7983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40949
I20260927 12:35:51.968463 8113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40949 every 8 connection(s)
I20260927 12:35:51.969508 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:51.972084 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:51.957859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37285 (local address 127.0.98.62:39247)
0927 12:35:51.958060 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:51.958065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:51.959040 (+ 975us) server_negotiation.cc:407] Connection header received
0927 12:35:51.960323 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:51.960327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:51.960389 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:51.960490 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:51.962193 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.962877 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.964642 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.964946 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.965051 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:51.968295 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:51.968299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:51.968302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:51.968335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:51.969918 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:51.970547 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:51.970551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:51.970553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:51.970634 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:51.970966 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:51.970969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:51.970971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:51.971165 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:51.971251 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:51.971771 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 12:35:51.971916 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 12:35:51.972921 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:51.958328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.98.1:37285)
0927 12:35:51.958847 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:51.958906 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:51.960044 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:51.960728 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:51.960739 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:51.961259 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:51.961976 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:51.962005 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.963002 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.963006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:51.964508 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:51.964521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.965113 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.965119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:51.965322 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:51.966021 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:51.966043 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:51.967975 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:51.970066 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:51.970072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:51.970088 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:51.970428 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:51.970720 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:51.970723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:51.970726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:51.970863 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:51.971264 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:51.971271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:51.971531 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 12:35:51.971694 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 12:35:51.972037 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
I20260927 12:35:51.974324 8114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39247
I20260927 12:35:51.974495 8114 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:51.975170 8114 heartbeater.cc:511] Master 127.0.98.62:39247 requested a full tablet report, sending...
I20260927 12:35:51.977023 7911 ts_manager.cc:194] Registered new tserver with Master: 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949)
I20260927 12:35:51.978507 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37285
I20260927 12:35:51.978564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 7983
I20260927 12:35:51.978675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:51.979028 392 external_mini_cluster.cc:1428] Setting key 5547e0c28b713e6c4cffc3bedf65ea3d
I20260927 12:35:51.985168 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:51.976864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.98.1:40345)
0927 12:35:51.977369 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:51.977385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:51.977494 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:51.977727 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:51.977730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:51.977957 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:51.978088 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:51.978095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.979190 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.979194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:51.979929 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:51.979941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.980620 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.980624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:51.980754 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:51.981328 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:51.981341 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:51.981689 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:51.983304 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:51.983308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:51.983310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:51.983547 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:51.983904 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:51.983908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:51.983910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:51.984025 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:51.984509 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:51.984512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:51.984633 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 12:35:51.984743 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:35:51.984945 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":10,"thread_start_us":201,"threads_started":1}
I20260927 12:35:51.985131 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:51.976973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40345 (local address 127.0.98.62:39247)
0927 12:35:51.977140 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:51.977145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:51.977477 (+ 332us) server_negotiation.cc:407] Connection header received
0927 12:35:51.977575 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:51.977577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:51.977619 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:51.977691 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:51.978219 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.979000 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.980052 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:51.980392 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:51.980495 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:51.981847 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:51.981852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:51.981854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:51.981877 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:51.983173 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:51.983679 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:51.983682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:51.983684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:51.983796 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:51.984186 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:51.984189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:51.984190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:51.984411 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:51.984502 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:51.984877 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 12:35:51.985005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":76}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 12:35:52.012970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39247
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:52.134819 8123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:52.135118 8123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:52.135164 8123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:52.139564 8123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:52.139645 8123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:52.139686 8123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:52.139720 8123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:52.145473 8123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:52.146625 8123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:52.147675 8123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:52.157387 8129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:52.157445 8123 server_base.cc:1034] running on GCE node
W20260927 12:35:52.157487 8128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:52.157507 8131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:52.157998 8123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:52.158638 8123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:52.159839 8123 hybrid_clock.cc:648] HybridClock initialized: now 1790512552159809 us; error 43 us; skew 500 ppm
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512552, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:52.162823 8123 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:35:52.163807 8123 webserver.cc:460] Webserver started at http://127.0.98.2:34809/ using document root <none> and password file <none>
I20260927 12:35:52.164453 8123 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:52.164522 8123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:52.164721 8123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:52.166865 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea8388b5de5b4350a87d2f4aec27c491"
format_stamp: "Formatted at 2026-09-27 12:35:52 on dist-test-slave-tzgx"
server_key: "25d525657718840e49e8aeda2ceb44b0"
server_key_iv: "ce2e817fb75cce90d5c8e1af40f3d285"
server_key_version: "encryptionkey@0"
I20260927 12:35:52.167434 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea8388b5de5b4350a87d2f4aec27c491"
format_stamp: "Formatted at 2026-09-27 12:35:52 on dist-test-slave-tzgx"
server_key: "25d525657718840e49e8aeda2ceb44b0"
server_key_iv: "ce2e817fb75cce90d5c8e1af40f3d285"
server_key_version: "encryptionkey@0"
I20260927 12:35:52.171401 8123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:52.174245 8138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:52.175778 8123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 12:35:52.176059 8123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "ea8388b5de5b4350a87d2f4aec27c491"
format_stamp: "Formatted at 2026-09-27 12:35:52 on dist-test-slave-tzgx"
server_key: "25d525657718840e49e8aeda2ceb44b0"
server_key_iv: "ce2e817fb75cce90d5c8e1af40f3d285"
server_key_version: "encryptionkey@0"
I20260927 12:35:52.176246 8123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:52.190471 8123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:52.194161 8123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:52.194360 8123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:52.196235 8123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:52.196303 8123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:52.196350 8123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:52.196378 8123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512552, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:52.207677 8123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39587
I20260927 12:35:52.208258 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39587 every 8 connection(s)
I20260927 12:35:52.208702 8123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:52.210249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 8123
I20260927 12:35:52.210346 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.197212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42637 (local address 127.0.98.62:39247)
0927 12:35:52.197351 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.197355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.198547 (+ 1192us) server_negotiation.cc:407] Connection header received
0927 12:35:52.199815 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.199854 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.199929 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.200016 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.201484 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.202089 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.203437 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.203754 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.203901 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.206612 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:52.206616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:52.206618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:52.206644 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:52.208175 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.208690 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.208694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.208696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.208791 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.209094 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.209098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.209100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.209298 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:52.209383 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.209987 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.210150 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:35:52.210362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:52.210636 392 external_mini_cluster.cc:1428] Setting key 0fff0f4f5d32ae2463c284f006c16e9a
I20260927 12:35:52.211151 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.197770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.98.2:42637)
0927 12:35:52.198350 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.198418 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.199471 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.200258 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.200268 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.200676 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:52.201316 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.201345 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.202218 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.202221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.203273 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.203284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.203963 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.203967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.204208 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.204912 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:52.204929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:52.206427 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:52.208289 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.208295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.208311 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.208591 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.208867 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.208870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.208872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.209009 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.209396 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:52.209403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:52.209733 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.209930 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.210262 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":145,"threads_started":1}
I20260927 12:35:52.212589 8254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39247
I20260927 12:35:52.212723 8254 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:52.213333 8254 heartbeater.cc:511] Master 127.0.98.62:39247 requested a full tablet report, sending...
I20260927 12:35:52.214414 7910 ts_manager.cc:194] Registered new tserver with Master: ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:52.215111 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42637
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:35:52.222523 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.214691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.98.2:52737)
0927 12:35:52.215028 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.215043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.215142 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.215495 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.215498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.215686 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:52.215837 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.215846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.216713 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.216716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.217317 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.217325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.217742 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.217745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.217837 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.218382 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:52.218393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:52.218721 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:52.220754 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.220758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.220760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.220966 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.221241 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.221244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.221246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.221331 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.221846 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:52.221849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:52.221950 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.222149 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.222345 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260927 12:35:52.222528 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.214755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52737 (local address 127.0.98.62:39247)
0927 12:35:52.214862 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.214867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.215102 (+ 235us) server_negotiation.cc:407] Connection header received
0927 12:35:52.215247 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.215251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.215306 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.215357 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.215971 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.216560 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.217422 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.217643 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.217729 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.218905 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:52.218908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:52.218910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:52.218933 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:52.220639 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.221068 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.221072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.221074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.221155 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.221409 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.221413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.221415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.221749 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:52.221847 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.222270 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.222393 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:35:52.241374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39247
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:52.358263 8263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:52.358543 8263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:52.358590 8263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:52.362828 8263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:52.362895 8263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:52.362934 8263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:35:52.362967 8263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:52.368444 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:52.369509 8263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:52.370488 8263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:52.380550 8271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:52.380709 8269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:52.380550 8268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:52.381031 8263 server_base.cc:1034] running on GCE node
I20260927 12:35:52.381435 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:52.382112 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:52.383288 8263 hybrid_clock.cc:648] HybridClock initialized: now 1790512552383266 us; error 34 us; skew 500 ppm
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512552, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:52.386411 8263 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:35:52.387528 8263 webserver.cc:460] Webserver started at http://127.0.98.3:44363/ using document root <none> and password file <none>
I20260927 12:35:52.388176 8263 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:52.388253 8263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:52.388463 8263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:52.390560 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "0ca452e86239484bac1556d74fe334e6"
format_stamp: "Formatted at 2026-09-27 12:35:52 on dist-test-slave-tzgx"
server_key: "b17af64eb7a4c54e68cf8d3d5ba9f1f5"
server_key_iv: "d4b0bcc31eaf3d310d0e5603a762baf9"
server_key_version: "encryptionkey@0"
I20260927 12:35:52.391160 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0ca452e86239484bac1556d74fe334e6"
format_stamp: "Formatted at 2026-09-27 12:35:52 on dist-test-slave-tzgx"
server_key: "b17af64eb7a4c54e68cf8d3d5ba9f1f5"
server_key_iv: "d4b0bcc31eaf3d310d0e5603a762baf9"
server_key_version: "encryptionkey@0"
I20260927 12:35:52.395704 8263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:35:52.398588 8278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:52.400273 8263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:52.400416 8263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "0ca452e86239484bac1556d74fe334e6"
format_stamp: "Formatted at 2026-09-27 12:35:52 on dist-test-slave-tzgx"
server_key: "b17af64eb7a4c54e68cf8d3d5ba9f1f5"
server_key_iv: "d4b0bcc31eaf3d310d0e5603a762baf9"
server_key_version: "encryptionkey@0"
I20260927 12:35:52.400554 8263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:52.411860 8263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:52.415364 8263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:52.415562 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:52.417140 8263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:52.417191 8263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:52.417245 8263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:52.417270 8263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:52.427178 8263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44047
I20260927 12:35:52.427238 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44047 every 8 connection(s)
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512552, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:52.428200 8263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:52.431559 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.417853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57815 (local address 127.0.98.62:39247)
0927 12:35:52.418029 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.418034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.418944 (+ 910us) server_negotiation.cc:407] Connection header received
0927 12:35:52.420137 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.420141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.420205 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.420292 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.422169 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.423005 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.424873 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.425157 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.425284 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.428038 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:52.428041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:52.428044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:52.428072 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:52.429603 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.430085 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.430089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.430091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.430173 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.430481 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.430485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.430487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.430675 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:52.430760 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.431263 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.431395 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:35:52.432440 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.418314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.98.3:57815)
0927 12:35:52.418776 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.418825 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.419816 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.420638 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.420647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.421163 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:52.421963 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.421989 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.423155 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.423160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.424707 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.424722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.425450 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.425454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.425612 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.426247 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:52.426265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:52.427837 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:52.429719 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.429725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.429740 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.429991 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.430253 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.430256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.430258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.430390 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.430771 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:52.430778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:52.431026 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.431193 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.431563 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260927 12:35:52.433732 8394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39247
I20260927 12:35:52.433930 8394 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:52.434448 8394 heartbeater.cc:511] Master 127.0.98.62:39247 requested a full tablet report, sending...
I20260927 12:35:52.435575 7911 ts_manager.cc:194] Registered new tserver with Master: 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047)
I20260927 12:35:52.436275 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57815
I20260927 12:35:52.438380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 8263
I20260927 12:35:52.438481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:52.438746 392 external_mini_cluster.cc:1428] Setting key 9b50dc649d8eef6442e5a7177183dbdf
I20260927 12:35:52.441005 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:35:52.443080 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.435822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.98.3:52985)
0927 12:35:52.436186 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.436197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.436279 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.436512 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.436515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.436724 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:52.436868 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.436875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.437722 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.437724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.438283 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.438291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.438772 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.438776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.438868 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.439391 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:52.439405 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:52.439788 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:52.441633 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.441637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.441639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.441867 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.442111 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.442114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.442116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.442202 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.442539 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:52.442541 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:52.442642 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.442746 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.442910 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":86,"threads_started":1}
I20260927 12:35:52.443080 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.435903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52985 (local address 127.0.98.62:39247)
0927 12:35:52.436023 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.436027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.436259 (+ 232us) server_negotiation.cc:407] Connection header received
0927 12:35:52.436361 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.436365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.436409 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.436492 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.436982 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.437597 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.438389 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.438671 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.438757 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.439971 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:52.439974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:52.439977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:52.440005 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:52.441530 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.441965 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.441968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.441969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.442040 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.442275 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.442277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.442278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.442465 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:52.442537 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.442807 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.442928 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:35:52.465462 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:52.459340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.98.62:39247)
0927 12:35:52.459495 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.459499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.459512 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:52.459550 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.459552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.459588 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.459657 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.460413 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.460937 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.461773 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.461988 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.462078 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.465345 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52252: BlockingRecv error: recv got EOF from 127.0.0.1:52252 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:52.475435 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.467204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.0.98.62:39247)
0927 12:35:52.467338 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.467343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.467444 (+ 101us) server_negotiation.cc:407] Connection header received
0927 12:35:52.467547 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.467549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.467589 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.467657 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.468358 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.468872 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.469720 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.469941 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.470049 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.472499 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:52.472502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:52.472504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:52.472531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:52.473846 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.474261 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.474265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.474267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.474352 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.474584 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.474586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.474587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.474816 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:52.474917 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.475149 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.475247 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:52.478602 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:52.481863 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:52.496879 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.491556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.0.98.1:40949)
0927 12:35:52.492069 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.492075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.492120 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:35:52.492201 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.492206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.492428 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.492932 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.493071 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.493891 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.495254 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.496177 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.496289 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.496487 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.496615 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.496722 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":78,"threads_started":1}
I20260927 12:35:52.496899 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.490895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.0.1:55762)
0927 12:35:52.491399 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.491450 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.491594 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.492591 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.492596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.492617 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.492761 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.492767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.494000 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.494003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.495131 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.495141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.496286 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.496290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.496410 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.496469 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.496584 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.496724 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"spinlock_wait_cycles":4480,"thread_start_us":77,"threads_started":1}
I20260927 12:35:52.497427 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.491001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44047 (local address 127.0.0.1:53436)
0927 12:35:52.491399 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.491450 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.491594 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.493078 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.493081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.493094 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.493237 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.493245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.494600 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.494603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.495728 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.495738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.496899 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.496903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.497135 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.497161 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.497216 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.497290 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":45,"spinlock_wait_cycles":10368,"thread_start_us":113,"threads_started":1}
I20260927 12:35:52.497679 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.491589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39587 (local address 127.0.0.1:53472)
0927 12:35:52.492422 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.492435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.492518 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.493003 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.493006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.493024 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.493148 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.493155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.495275 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.495277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.496246 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.496257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.497318 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.497321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.497404 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.497425 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.497484 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.497553 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":30,"thread_start_us":100,"threads_started":1}
I20260927 12:35:52.497748 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.492024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.98.2:39587)
0927 12:35:52.492480 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.492492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.492515 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:35:52.492657 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.492663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.492860 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.492994 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.493271 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.494470 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.496399 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.497203 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.497318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.497483 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.497533 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.497624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":146,"threads_started":1}
I20260927 12:35:52.498534 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.492010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.0.98.3:44047)
0927 12:35:52.492702 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.492713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.492734 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:35:52.492793 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.492796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.492934 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.493183 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.493319 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.494470 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.495894 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.496749 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.497668 (+ 919us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.497946 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.498000 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.498395 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":145,"threads_started":1}
I20260927 12:35:52.500277 8048 tablet_service.cc:1467] Processing CreateTablet for tablet d1703392f7104776800a714e40fa3674 (DEFAULT_TABLE table=test-table [id=3a2ddc74c76f4cfaae23a86c638c952f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:52.501272 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1703392f7104776800a714e40fa3674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.501120 8188 tablet_service.cc:1467] Processing CreateTablet for tablet d1703392f7104776800a714e40fa3674 (DEFAULT_TABLE table=test-table [id=3a2ddc74c76f4cfaae23a86c638c952f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:52.501590 8328 tablet_service.cc:1467] Processing CreateTablet for tablet d1703392f7104776800a714e40fa3674 (DEFAULT_TABLE table=test-table [id=3a2ddc74c76f4cfaae23a86c638c952f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:52.502211 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1703392f7104776800a714e40fa3674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.502692 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1703392f7104776800a714e40fa3674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.508610 8422 tablet_bootstrap.cc:492] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Bootstrap starting.
I20260927 12:35:52.510318 8424 tablet_bootstrap.cc:492] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Bootstrap starting.
I20260927 12:35:52.511659 8422 tablet_bootstrap.cc:654] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.512465 8423 tablet_bootstrap.cc:492] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Bootstrap starting.
I20260927 12:35:52.512626 8424 tablet_bootstrap.cc:654] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.512651 8422 log.cc:824] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:52.513473 8424 log.cc:824] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:52.514551 8423 tablet_bootstrap.cc:654] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.514897 8422 tablet_bootstrap.cc:492] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: No bootstrap required, opened a new log
I20260927 12:35:52.515110 8422 ts_tablet_manager.cc:1397] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 12:35:52.515249 8423 log.cc:824] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:52.515520 8424 tablet_bootstrap.cc:492] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: No bootstrap required, opened a new log
I20260927 12:35:52.515715 8424 ts_tablet_manager.cc:1397] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:35:52.518131 8423 tablet_bootstrap.cc:492] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: No bootstrap required, opened a new log
I20260927 12:35:52.518334 8423 ts_tablet_manager.cc:1397] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 12:35:52.519680 8424 raft_consensus.cc:348] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.519680 8422 raft_consensus.cc:348] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.519982 8422 raft_consensus.cc:374] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.519981 8424 raft_consensus.cc:374] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.520042 8424 raft_consensus.cc:729] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8388b5de5b4350a87d2f4aec27c491, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.520042 8422 raft_consensus.cc:729] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.520552 8422 consensus_queue.cc:260] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.520552 8424 consensus_queue.cc:260] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.521284 8114 heartbeater.cc:503] Master 127.0.98.62:39247 was elected leader, sending a full tablet report...
I20260927 12:35:52.521620 8422 ts_tablet_manager.cc:1428] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:35:52.521620 8424 ts_tablet_manager.cc:1428] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:35:52.521955 8254 heartbeater.cc:503] Master 127.0.98.62:39247 was elected leader, sending a full tablet report...
I20260927 12:35:52.521916 8423 raft_consensus.cc:348] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.522192 8423 raft_consensus.cc:374] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.522243 8423 raft_consensus.cc:729] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.522688 8423 consensus_queue.cc:260] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.523808 8423 ts_tablet_manager.cc:1428] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:35:52.524251 8394 heartbeater.cc:503] Master 127.0.98.62:39247 was elected leader, sending a full tablet report...
I20260927 12:35:52.562153 8428 raft_consensus.cc:482] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:52.562366 8428 raft_consensus.cc:504] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.563484 8428 leader_election.cc:288] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047), ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:52.569242 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.563900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41973 (local address 127.0.98.3:44047)
0927 12:35:52.564019 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.564024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.564259 (+ 235us) server_negotiation.cc:407] Connection header received
0927 12:35:52.564431 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.564435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.564486 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.564538 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.564975 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.565784 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.567868 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.568538 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.568664 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.568737 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.569021 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.569112 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:35:52.569242 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.564313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59799 (local address 127.0.98.2:39587)
0927 12:35:52.564435 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.564439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.564681 (+ 242us) server_negotiation.cc:407] Connection header received
0927 12:35:52.564742 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.564744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.564788 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.564871 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.565102 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.565916 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.567868 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.568639 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.568731 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.568805 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.569026 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.569112 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:35:52.569268 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.564193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39587 (local address 127.0.98.1:59799)
0927 12:35:52.564553 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.564565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.564669 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.564893 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.564896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.564911 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.565004 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.565009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.566021 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.566024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.567699 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.567714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.568743 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.568747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.568878 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.568895 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.568996 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.569049 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":172,"thread_start_us":121,"threads_started":1}
I20260927 12:35:52.569268 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.563779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44047 (local address 127.0.98.1:41973)
0927 12:35:52.564155 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.564170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.564300 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.564681 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.564684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.564696 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.564812 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.564817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.565938 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.565941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.567702 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.567714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.568689 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.568694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.568806 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.568865 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.568950 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.569050 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":123,"thread_start_us":117,"threads_started":1}
I20260927 12:35:52.570007 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1703392f7104776800a714e40fa3674" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491" is_pre_election: true
I20260927 12:35:52.570080 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1703392f7104776800a714e40fa3674" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca452e86239484bac1556d74fe334e6" is_pre_election: true
I20260927 12:35:52.570322 8208 raft_consensus.cc:2454] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 0.
I20260927 12:35:52.570363 8348 raft_consensus.cc:2454] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 0.
I20260927 12:35:52.570796 8002 leader_election.cc:302] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c57a90c244645e9a6ad7d1533fd983c, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:52.571115 8428 raft_consensus.cc:2781] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:52.571214 8428 raft_consensus.cc:482] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:52.571256 8428 raft_consensus.cc:3037] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.572333 8428 raft_consensus.cc:504] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.572690 8428 leader_election.cc:288] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 election: Requested vote from peers 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047), ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:52.573110 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1703392f7104776800a714e40fa3674" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491"
I20260927 12:35:52.573135 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1703392f7104776800a714e40fa3674" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca452e86239484bac1556d74fe334e6"
I20260927 12:35:52.573248 8208 raft_consensus.cc:3037] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.573276 8348 raft_consensus.cc:3037] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.574355 8208 raft_consensus.cc:2454] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 1.
I20260927 12:35:52.574445 8348 raft_consensus.cc:2454] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 1.
I20260927 12:35:52.574750 8002 leader_election.cc:302] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c57a90c244645e9a6ad7d1533fd983c, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:52.574976 8428 raft_consensus.cc:2781] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:52.575197 8428 raft_consensus.cc:686] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 LEADER]: Becoming Leader. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Running, Role: LEADER
I20260927 12:35:52.575539 8428 consensus_queue.cc:237] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } }
I20260927 12:35:52.578912 7909 catalog_manager.cc:5495] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:52.602468 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.598766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.0.98.1:40949)
0927 12:35:52.598977 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.598982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.598999 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:35:52.599091 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.599093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.599150 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.599243 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:52.599505 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.600025 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.600922 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.601149 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.601241 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.601560 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:52.602052 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:52.602162 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.602249 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.602316 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 12:35:52.609925 8348 raft_consensus.cc:1261] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c57a90c244645e9a6ad7d1533fd983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:52.609958 8208 raft_consensus.cc:1261] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Refusing update from remote peer 6c57a90c244645e9a6ad7d1533fd983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:52.610715 8433 consensus_queue.cc:1035] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:52.610924 8428 consensus_queue.cc:1035] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:52.618635 8436 mvcc.cc:205] Tried to move back new op lower bound from 7333939415481819136 to 7333939415352475648. Current Snapshot: MvccSnapshot[applied={T|T < 7333939415481819136}]
I20260927 12:35:52.630234 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:52.630463 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:52.633536 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Sending DeleteTablet for 3 replicas of tablet d1703392f7104776800a714e40fa3674
I20260927 12:35:52.634609 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet d1703392f7104776800a714e40fa3674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260927 12:35:52.634613 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet d1703392f7104776800a714e40fa3674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260927 12:35:52.634747 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet d1703392f7104776800a714e40fa3674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53436
I20260927 12:35:52.635293 8448 tablet_replica.cc:330] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: stopping tablet replica
I20260927 12:35:52.635704 8447 tablet_replica.cc:330] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: stopping tablet replica
I20260927 12:35:52.635715 8448 raft_consensus.cc:2229] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:52.636039 8447 raft_consensus.cc:2229] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:52.636081 8448 raft_consensus.cc:2258] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:52.636024 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:35:52.636490 8447 raft_consensus.cc:2258] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:35:52.636552 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:52.638168 8446 tablet_replica.cc:330] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: stopping tablet replica
I20260927 12:35:52.638514 8446 raft_consensus.cc:2229] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:52.638837 8446 raft_consensus.cc:2258] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:52.639514 8448 ts_tablet_manager.cc:1905] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:52.640415 8447 ts_tablet_manager.cc:1905] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:52.642204 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 2cccc1be3dad470fac543a719b5e7a5c (DEFAULT_TABLE table=test-table [id=52100d83d84645c994827fab097f16d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:52.642204 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 2cccc1be3dad470fac543a719b5e7a5c (DEFAULT_TABLE table=test-table [id=52100d83d84645c994827fab097f16d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:52.642473 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cccc1be3dad470fac543a719b5e7a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.642472 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cccc1be3dad470fac543a719b5e7a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.644201 8446 ts_tablet_manager.cc:1905] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:52.644744 8423 tablet_bootstrap.cc:492] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Bootstrap starting.
I20260927 12:35:52.644820 8422 tablet_bootstrap.cc:492] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Bootstrap starting.
I20260927 12:35:52.642475 8448 ts_tablet_manager.cc:1918] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:52.645584 8448 log.cc:1196] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/d1703392f7104776800a714e40fa3674
I20260927 12:35:52.645790 8422 tablet_bootstrap.cc:654] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.645807 8423 tablet_bootstrap.cc:654] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.646116 8448 ts_tablet_manager.cc:1939] T d1703392f7104776800a714e40fa3674 P 0ca452e86239484bac1556d74fe334e6: Deleting consensus metadata
I20260927 12:35:52.647233 8446 ts_tablet_manager.cc:1918] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:52.647349 8446 log.cc:1196] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/d1703392f7104776800a714e40fa3674
I20260927 12:35:52.647783 8446 ts_tablet_manager.cc:1939] T d1703392f7104776800a714e40fa3674 P ea8388b5de5b4350a87d2f4aec27c491: Deleting consensus metadata
I20260927 12:35:52.647821 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 2cccc1be3dad470fac543a719b5e7a5c (DEFAULT_TABLE table=test-table [id=52100d83d84645c994827fab097f16d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:52.648114 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cccc1be3dad470fac543a719b5e7a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.649983 8423 tablet_bootstrap.cc:492] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: No bootstrap required, opened a new log
I20260927 12:35:52.650100 8423 ts_tablet_manager.cc:1397] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 12:35:52.650432 8422 tablet_bootstrap.cc:492] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: No bootstrap required, opened a new log
I20260927 12:35:52.650542 8422 ts_tablet_manager.cc:1397] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 12:35:52.650772 8423 raft_consensus.cc:348] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.650909 8423 raft_consensus.cc:374] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.650905 8424 tablet_bootstrap.cc:492] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Bootstrap starting.
I20260927 12:35:52.650952 8423 raft_consensus.cc:729] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.651062 8422 raft_consensus.cc:348] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.651132 8423 consensus_queue.cc:260] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.651187 8422 raft_consensus.cc:374] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.651228 8422 raft_consensus.cc:729] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.651412 8423 ts_tablet_manager.cc:1428] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 12:35:52.651695 8447 ts_tablet_manager.cc:1918] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:52.651705 7898 catalog_manager.cc:4841] TS ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587): tablet d1703392f7104776800a714e40fa3674 (table test-table [id=3a2ddc74c76f4cfaae23a86c638c952f]) successfully deleted
I20260927 12:35:52.651875 8447 log.cc:1196] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/d1703392f7104776800a714e40fa3674
I20260927 12:35:52.651896 7898 catalog_manager.cc:4841] TS 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047): tablet d1703392f7104776800a714e40fa3674 (table test-table [id=3a2ddc74c76f4cfaae23a86c638c952f]) successfully deleted
I20260927 12:35:52.651979 8424 tablet_bootstrap.cc:654] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.652070 8422 consensus_queue.cc:260] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.652400 8447 ts_tablet_manager.cc:1939] T d1703392f7104776800a714e40fa3674 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting consensus metadata
I20260927 12:35:52.652848 8422 ts_tablet_manager.cc:1428] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 12:35:52.654450 8424 tablet_bootstrap.cc:492] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: No bootstrap required, opened a new log
I20260927 12:35:52.654547 8424 ts_tablet_manager.cc:1397] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:35:52.655042 8424 raft_consensus.cc:348] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.655161 8424 raft_consensus.cc:374] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.655190 8424 raft_consensus.cc:729] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8388b5de5b4350a87d2f4aec27c491, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.655303 8424 consensus_queue.cc:260] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.655568 8424 ts_tablet_manager.cc:1428] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:35:52.656428 7896 catalog_manager.cc:4841] TS 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949): tablet d1703392f7104776800a714e40fa3674 (table test-table [id=3a2ddc74c76f4cfaae23a86c638c952f]) successfully deleted
W20260927 12:35:52.679193 8395 tablet.cc:2369] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:52.694972 8429 raft_consensus.cc:482] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:52.695149 8429 raft_consensus.cc:504] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.696393 8429 leader_election.cc:288] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949), 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047)
I20260927 12:35:52.700874 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.696590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.98.2:42695)
0927 12:35:52.696845 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.696859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.696949 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.697200 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.697203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.697221 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.697335 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.697340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.698101 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.698105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.699463 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.699474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.700393 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.700397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.700516 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.700544 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.700629 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.700715 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":26}
I20260927 12:35:52.700872 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.696684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42695 (local address 127.0.98.1:40949)
0927 12:35:52.696807 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.696811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.697017 (+ 206us) server_negotiation.cc:407] Connection header received
0927 12:35:52.697074 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.697076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.697116 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.697174 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.697449 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.697994 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.699605 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.700236 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.700343 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.700412 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.700664 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.700745 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:35:52.700896 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.696926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51489 (local address 127.0.98.3:44047)
0927 12:35:52.697052 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.697057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.697254 (+ 197us) server_negotiation.cc:407] Connection header received
0927 12:35:52.697460 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.697463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.697508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.697554 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:52.697956 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.698496 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.699581 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.700236 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.700343 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.700409 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.700709 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.700782 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:35:52.701321 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.696640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44047 (local address 127.0.98.2:51489)
0927 12:35:52.697174 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.697187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.697340 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.697679 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.697682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.697698 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:52.697825 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.697832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.698608 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.698612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.699458 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.699468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.700495 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.700499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.700575 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.700593 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.701099 (+ 506us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.701177 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":26,"thread_start_us":69,"threads_started":1}
I20260927 12:35:52.701615 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cccc1be3dad470fac543a719b5e7a5c" candidate_uuid: "ea8388b5de5b4350a87d2f4aec27c491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57a90c244645e9a6ad7d1533fd983c" is_pre_election: true
I20260927 12:35:52.701742 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cccc1be3dad470fac543a719b5e7a5c" candidate_uuid: "ea8388b5de5b4350a87d2f4aec27c491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca452e86239484bac1556d74fe334e6" is_pre_election: true
I20260927 12:35:52.701875 8068 raft_consensus.cc:2454] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8388b5de5b4350a87d2f4aec27c491 in term 0.
I20260927 12:35:52.701897 8348 raft_consensus.cc:2454] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8388b5de5b4350a87d2f4aec27c491 in term 0.
I20260927 12:35:52.702243 8142 leader_election.cc:302] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:52.702512 8429 raft_consensus.cc:2781] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:52.702580 8429 raft_consensus.cc:482] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:52.702625 8429 raft_consensus.cc:3037] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.703998 8429 raft_consensus.cc:504] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.704537 8429 leader_election.cc:288] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [CANDIDATE]: Term 1 election: Requested vote from peers 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949), 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047)
I20260927 12:35:52.704910 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cccc1be3dad470fac543a719b5e7a5c" candidate_uuid: "ea8388b5de5b4350a87d2f4aec27c491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca452e86239484bac1556d74fe334e6"
I20260927 12:35:52.705046 8348 raft_consensus.cc:3037] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.706224 8348 raft_consensus.cc:2454] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8388b5de5b4350a87d2f4aec27c491 in term 1.
I20260927 12:35:52.706496 8142 leader_election.cc:302] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:52.706525 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cccc1be3dad470fac543a719b5e7a5c" candidate_uuid: "ea8388b5de5b4350a87d2f4aec27c491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57a90c244645e9a6ad7d1533fd983c"
I20260927 12:35:52.706655 8068 raft_consensus.cc:3037] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.706682 8429 raft_consensus.cc:2781] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:52.706840 8429 raft_consensus.cc:686] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 1 LEADER]: Becoming Leader. State: Replica: ea8388b5de5b4350a87d2f4aec27c491, State: Running, Role: LEADER
I20260927 12:35:52.707113 8429 consensus_queue.cc:237] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.707746 8068 raft_consensus.cc:2454] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8388b5de5b4350a87d2f4aec27c491 in term 1.
W20260927 12:35:52.709875 8255 tablet.cc:2369] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:35:52.709736 7911 catalog_manager.cc:5495] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ea8388b5de5b4350a87d2f4aec27c491" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } health_report { overall_health: UNKNOWN } } }
W20260927 12:35:52.720966 8115 tablet.cc:2369] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:35:52 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512551, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:52.750751 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.741897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39247 (local address 127.0.0.1:52260)
0927 12:35:52.742185 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.742197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.742310 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.742525 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.742528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.742939 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:52.743036 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.743043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.743779 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.743783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.744402 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.744412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.744803 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.744807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.744931 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.745560 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:52.745573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:52.747127 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:52.748893 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.748905 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.748912 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.749238 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.749522 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:52.749526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:52.749528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:52.749635 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:52.750006 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:52.750013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:52.750151 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.750280 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.750483 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":36,"thread_start_us":124,"threads_started":1}
I20260927 12:35:52.750749 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.741978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.0.98.62:39247)
0927 12:35:52.742104 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.742109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.742272 (+ 163us) server_negotiation.cc:407] Connection header received
0927 12:35:52.742396 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.742399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.742436 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.742499 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:52.743155 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.743684 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.744507 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.744708 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.744791 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.747296 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:52.747299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:52.747302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:52.747351 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:52.748769 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.749353 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.749356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.749357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.749429 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:52.749717 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:52.749720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:52.749721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:52.749901 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:52.749971 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.750312 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.750580 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":49}
I20260927 12:35:52.755210 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:35:52.756029 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:52.761404 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 73828e77ed794973a07978b0a4e4864b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=482c6a402936487188e2f9296dde01f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:52.761404 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 73828e77ed794973a07978b0a4e4864b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=482c6a402936487188e2f9296dde01f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:52.761494 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 73828e77ed794973a07978b0a4e4864b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=482c6a402936487188e2f9296dde01f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:35:52.761830 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73828e77ed794973a07978b0a4e4864b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.761830 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73828e77ed794973a07978b0a4e4864b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.761915 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73828e77ed794973a07978b0a4e4864b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:52.764766 8422 tablet_bootstrap.cc:492] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: Bootstrap starting.
I20260927 12:35:52.765015 8423 tablet_bootstrap.cc:492] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: Bootstrap starting.
I20260927 12:35:52.765327 8424 tablet_bootstrap.cc:492] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: Bootstrap starting.
I20260927 12:35:52.765849 8422 tablet_bootstrap.cc:654] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.765877 8423 tablet_bootstrap.cc:654] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.766340 8424 tablet_bootstrap.cc:654] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:52.767699 8422 tablet_bootstrap.cc:492] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: No bootstrap required, opened a new log
I20260927 12:35:52.767719 8424 tablet_bootstrap.cc:492] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: No bootstrap required, opened a new log
I20260927 12:35:52.767787 8422 ts_tablet_manager.cc:1397] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 12:35:52.767802 8424 ts_tablet_manager.cc:1397] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:52.767925 8423 tablet_bootstrap.cc:492] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: No bootstrap required, opened a new log
I20260927 12:35:52.767997 8423 ts_tablet_manager.cc:1397] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:52.768304 8422 raft_consensus.cc:348] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.768424 8422 raft_consensus.cc:374] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.768303 8424 raft_consensus.cc:348] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.768455 8422 raft_consensus.cc:729] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.768412 8423 raft_consensus.cc:348] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.768477 8424 raft_consensus.cc:374] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.768517 8424 raft_consensus.cc:729] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8388b5de5b4350a87d2f4aec27c491, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.768520 8423 raft_consensus.cc:374] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:52.768553 8423 raft_consensus.cc:729] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Initialized, Role: FOLLOWER
I20260927 12:35:52.768584 8422 consensus_queue.cc:260] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.768653 8423 consensus_queue.cc:260] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.768901 8423 ts_tablet_manager.cc:1428] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:52.768855 8442 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.769042 8442 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.769115 8430 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.769256 8430 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.768858 8422 ts_tablet_manager.cc:1428] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:52.768647 8424 consensus_queue.cc:260] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.770036 8429 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.770162 8429 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.771719 8424 ts_tablet_manager.cc:1428] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:35:52.912181 8442 raft_consensus.cc:482] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:52.912326 8442 raft_consensus.cc:504] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.912714 8442 leader_election.cc:288] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587), 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047)
I20260927 12:35:52.913245 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73828e77ed794973a07978b0a4e4864b" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491" is_pre_election: true
I20260927 12:35:52.913345 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73828e77ed794973a07978b0a4e4864b" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca452e86239484bac1556d74fe334e6" is_pre_election: true
I20260927 12:35:52.913466 8208 raft_consensus.cc:2454] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 0.
I20260927 12:35:52.913465 8348 raft_consensus.cc:2454] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 0.
I20260927 12:35:52.913808 8002 leader_election.cc:302] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c57a90c244645e9a6ad7d1533fd983c, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:52.914011 8442 raft_consensus.cc:2781] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:52.914079 8442 raft_consensus.cc:482] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:52.914119 8442 raft_consensus.cc:3037] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.915153 8442 raft_consensus.cc:504] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.915482 8442 leader_election.cc:288] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 election: Requested vote from peers ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587), 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047)
I20260927 12:35:52.915956 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73828e77ed794973a07978b0a4e4864b" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491"
I20260927 12:35:52.915956 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73828e77ed794973a07978b0a4e4864b" candidate_uuid: "6c57a90c244645e9a6ad7d1533fd983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca452e86239484bac1556d74fe334e6"
I20260927 12:35:52.916122 8348 raft_consensus.cc:3037] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.916115 8208 raft_consensus.cc:3037] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:52.917222 8208 raft_consensus.cc:2454] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 1.
I20260927 12:35:52.917222 8348 raft_consensus.cc:2454] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c57a90c244645e9a6ad7d1533fd983c in term 1.
I20260927 12:35:52.917554 8002 leader_election.cc:302] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, 6c57a90c244645e9a6ad7d1533fd983c; no voters:
I20260927 12:35:52.917755 8442 raft_consensus.cc:2781] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:52.917860 8442 raft_consensus.cc:686] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [term 1 LEADER]: Becoming Leader. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Running, Role: LEADER
I20260927 12:35:52.918007 8442 consensus_queue.cc:237] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:52.918641 8428 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: TxnStatusTablet state changed. Reason: New leader 6c57a90c244645e9a6ad7d1533fd983c. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.918751 8428 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:52.919006 8465 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:35:52.919169 8465 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:35:52.919857 7911 catalog_manager.cc:5495] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } health_report { overall_health: UNKNOWN } } }
I20260927 12:35:52.935039 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.931218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.0.1:55774)
0927 12:35:52.931372 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.931387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.931486 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.931750 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.931753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.931760 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:52.931873 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.931879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.932734 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.932737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.933762 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.933774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.934201 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.934205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.934302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.934360 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:52.934713 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:52.934724 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.934784 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.934882 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 12:35:52.935418 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.931316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.0.98.1:40949)
0927 12:35:52.931425 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.931428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.931441 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:52.931621 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.931623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.931671 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.931730 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:52.931983 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.932632 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.933874 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.934102 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.934185 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.934526 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:52.934627 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:52.934701 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.935228 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.935300 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:35:52.938074 8348 raft_consensus.cc:1261] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c57a90c244645e9a6ad7d1533fd983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:52.938131 8208 raft_consensus.cc:1261] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Refusing update from remote peer 6c57a90c244645e9a6ad7d1533fd983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:52.938549 8442 consensus_queue.cc:1035] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:52.938717 8428 consensus_queue.cc:1035] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:52.940111 8429 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: TxnStatusTablet state changed. Reason: New leader 6c57a90c244645e9a6ad7d1533fd983c. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.940228 8429 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.940171 8430 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: TxnStatusTablet state changed. Reason: New leader 6c57a90c244645e9a6ad7d1533fd983c. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.940291 8430 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.941686 8442 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.941799 8442 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:52.942560 8438 mvcc.cc:205] Tried to move back new op lower bound from 7333939416830111744 to 7333939416753356800. Current Snapshot: MvccSnapshot[applied={T|T < 7333939416830111744}]
I20260927 12:35:52.942698 8430 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.942799 8430 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 0ca452e86239484bac1556d74fe334e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.943524 8433 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.943643 8433 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P 6c57a90c244645e9a6ad7d1533fd983c: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:35:52.944433 8429 tablet_replica.cc:439] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c57a90c244645e9a6ad7d1533fd983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } }
I20260927 12:35:52.944550 8429 tablet_replica.cc:442] T 73828e77ed794973a07978b0a4e4864b P ea8388b5de5b4350a87d2f4aec27c491: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:35:52.947365 8437 mvcc.cc:205] Tried to move back new op lower bound from 7333939416830111744 to 7333939416753356800. Current Snapshot: MvccSnapshot[applied={T|T < 7333939416830111744 or (T in {7333939416830111744})}]
I20260927 12:35:52.952234 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.948014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.98.2:39587)
0927 12:35:52.948142 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.948146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.948159 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:52.948242 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.948245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.948305 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.948409 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:52.948689 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.949205 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.950502 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.950782 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.950888 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.951212 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:52.951721 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:52.951840 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.951971 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.952039 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:35:52.965121 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.960322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.98.2:43751)
0927 12:35:52.960693 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:52.960708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:52.960818 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:52.961073 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:52.961076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:52.961084 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:52.961222 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.961228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.961931 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.961934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.963453 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:52.963467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.964013 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.964017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:52.964164 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.964231 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:52.964702 (+ 471us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:52.964718 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:35:52.964840 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 12:35:52.964940 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
I20260927 12:35:52.965061 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:52.960439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43751 (local address 127.0.98.1:40949)
0927 12:35:52.960566 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:52.960571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:52.960789 (+ 218us) server_negotiation.cc:407] Connection header received
0927 12:35:52.960912 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:52.960915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:52.960974 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:52.961051 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:52.961337 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.961833 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.963626 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:52.963894 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:52.963987 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:52.964444 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:52.964533 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:52.964607 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:52.964859 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 12:35:52.964936 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:35:52.975575 8068 raft_consensus.cc:1261] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Refusing update from remote peer ea8388b5de5b4350a87d2f4aec27c491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:52.976253 8453 consensus_queue.cc:1035] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:52.976387 8348 raft_consensus.cc:1261] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Refusing update from remote peer ea8388b5de5b4350a87d2f4aec27c491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:52.976858 8453 consensus_queue.cc:1035] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:53.004319 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.000579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39587 (local address 127.0.98.1:36783)
0927 12:35:53.000903 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:53.000927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:53.001038 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:53.001383 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:53.001386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:53.001395 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:53.001499 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.001504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.002287 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.002290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.002977 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.002987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.003464 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.003468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.003549 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.003599 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.003889 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.003896 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:35:53.003956 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:35:53.004054 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260927 12:35:53.004319 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.000684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36783 (local address 127.0.98.2:39587)
0927 12:35:53.000802 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.000807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.001027 (+ 220us) server_negotiation.cc:407] Connection header received
0927 12:35:53.001127 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.001130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.001187 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.001261 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:53.001601 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.002196 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.003088 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.003362 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.003452 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.003721 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.003815 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.003874 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.003988 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.004060 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:35:53.036500 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:53.036702 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:53.039310 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Sending DeleteTablet for 3 replicas of tablet 2cccc1be3dad470fac543a719b5e7a5c
I20260927 12:35:53.040048 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cccc1be3dad470fac543a719b5e7a5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260927 12:35:53.040370 8447 tablet_replica.cc:330] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: stopping tablet replica
I20260927 12:35:53.040473 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cccc1be3dad470fac543a719b5e7a5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260927 12:35:53.040503 8447 raft_consensus.cc:2229] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:53.040630 8447 raft_consensus.cc:2258] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.040654 8446 tablet_replica.cc:330] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: stopping tablet replica
I20260927 12:35:53.040759 8446 raft_consensus.cc:2229] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:53.041093 8446 raft_consensus.cc:2258] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.041952 8447 ts_tablet_manager.cc:1905] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.041987 8446 ts_tablet_manager.cc:1905] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.044428 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cccc1be3dad470fac543a719b5e7a5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53436
I20260927 12:35:53.044669 8448 tablet_replica.cc:330] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: stopping tablet replica
I20260927 12:35:53.044783 8448 raft_consensus.cc:2229] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:53.044906 8448 raft_consensus.cc:2258] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.045073 8447 ts_tablet_manager.cc:1918] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:53.045149 8447 log.cc:1196] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/2cccc1be3dad470fac543a719b5e7a5c
I20260927 12:35:53.045372 8447 ts_tablet_manager.cc:1939] T 2cccc1be3dad470fac543a719b5e7a5c P 6c57a90c244645e9a6ad7d1533fd983c: Deleting consensus metadata
I20260927 12:35:53.045791 8448 ts_tablet_manager.cc:1905] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.046509 7896 catalog_manager.cc:4841] TS 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949): tablet 2cccc1be3dad470fac543a719b5e7a5c (table test-table [id=52100d83d84645c994827fab097f16d9]) successfully deleted
I20260927 12:35:53.048753 8448 ts_tablet_manager.cc:1918] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:53.048847 8448 log.cc:1196] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/2cccc1be3dad470fac543a719b5e7a5c
I20260927 12:35:53.049729 8446 ts_tablet_manager.cc:1918] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:53.049868 8446 log.cc:1196] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/2cccc1be3dad470fac543a719b5e7a5c
I20260927 12:35:53.050292 8446 ts_tablet_manager.cc:1939] T 2cccc1be3dad470fac543a719b5e7a5c P ea8388b5de5b4350a87d2f4aec27c491: Deleting consensus metadata
I20260927 12:35:53.051975 7898 catalog_manager.cc:4841] TS ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587): tablet 2cccc1be3dad470fac543a719b5e7a5c (table test-table [id=52100d83d84645c994827fab097f16d9]) successfully deleted
I20260927 12:35:53.052546 8448 ts_tablet_manager.cc:1939] T 2cccc1be3dad470fac543a719b5e7a5c P 0ca452e86239484bac1556d74fe334e6: Deleting consensus metadata
I20260927 12:35:53.053562 7898 catalog_manager.cc:4841] TS 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047): tablet 2cccc1be3dad470fac543a719b5e7a5c (table test-table [id=52100d83d84645c994827fab097f16d9]) successfully deleted
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:35:53.083407 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:53.074768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.62:39247)
0927 12:35:53.074917 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.074922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.075146 (+ 224us) server_negotiation.cc:407] Connection header received
0927 12:35:53.075385 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.075389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.075449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.075572 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:53.076552 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.077348 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.078556 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.078922 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.079030 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.083269 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52276: BlockingRecv error: recv got EOF from 127.0.0.1:52276 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:35:53.095973 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.085340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.98.62:39247)
0927 12:35:53.085459 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.085463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.085666 (+ 203us) server_negotiation.cc:407] Connection header received
0927 12:35:53.085818 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.085821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.085869 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.085957 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:53.086704 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.087540 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.088769 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.089120 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.089223 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.092214 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:53.092218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:53.092222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:53.092258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:53.093900 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:53.094514 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:53.094518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:53.094521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:53.094636 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:53.094993 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:53.094997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:53.094999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:53.095251 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:53.095395 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.095657 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.095791 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:35:53.098742 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:35:53.099179 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:53.104282 8048 tablet_service.cc:1467] Processing CreateTablet for tablet d506191e8d964946a6e2c45826ad0a58 (DEFAULT_TABLE table=test-table [id=d324a881585040699f0e5ae95c101976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:53.104277 8328 tablet_service.cc:1467] Processing CreateTablet for tablet d506191e8d964946a6e2c45826ad0a58 (DEFAULT_TABLE table=test-table [id=d324a881585040699f0e5ae95c101976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:53.104599 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d506191e8d964946a6e2c45826ad0a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:53.104687 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d506191e8d964946a6e2c45826ad0a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:53.106930 8422 tablet_bootstrap.cc:492] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Bootstrap starting.
I20260927 12:35:53.107220 8188 tablet_service.cc:1467] Processing CreateTablet for tablet d506191e8d964946a6e2c45826ad0a58 (DEFAULT_TABLE table=test-table [id=d324a881585040699f0e5ae95c101976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:53.107493 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d506191e8d964946a6e2c45826ad0a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:53.107849 8422 tablet_bootstrap.cc:654] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:53.108321 8423 tablet_bootstrap.cc:492] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Bootstrap starting.
I20260927 12:35:53.109216 8423 tablet_bootstrap.cc:654] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:53.109664 8422 tablet_bootstrap.cc:492] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: No bootstrap required, opened a new log
I20260927 12:35:53.109771 8422 ts_tablet_manager.cc:1397] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:53.110406 8422 raft_consensus.cc:348] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.110551 8422 raft_consensus.cc:374] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:53.110582 8422 raft_consensus.cc:729] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Initialized, Role: FOLLOWER
I20260927 12:35:53.110754 8422 consensus_queue.cc:260] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.111797 8422 ts_tablet_manager.cc:1428] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:35:53.113062 8424 tablet_bootstrap.cc:492] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Bootstrap starting.
I20260927 12:35:53.114243 8424 tablet_bootstrap.cc:654] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:53.114295 8423 tablet_bootstrap.cc:492] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: No bootstrap required, opened a new log
I20260927 12:35:53.114383 8423 ts_tablet_manager.cc:1397] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 12:35:53.114868 8423 raft_consensus.cc:348] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.115006 8423 raft_consensus.cc:374] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:53.115042 8423 raft_consensus.cc:729] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Initialized, Role: FOLLOWER
I20260927 12:35:53.115180 8423 consensus_queue.cc:260] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.115487 8423 ts_tablet_manager.cc:1428] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:53.115767 8424 tablet_bootstrap.cc:492] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: No bootstrap required, opened a new log
I20260927 12:35:53.115849 8424 ts_tablet_manager.cc:1397] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:53.116295 8424 raft_consensus.cc:348] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.116391 8424 raft_consensus.cc:374] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:53.116421 8424 raft_consensus.cc:729] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8388b5de5b4350a87d2f4aec27c491, State: Initialized, Role: FOLLOWER
I20260927 12:35:53.116536 8424 consensus_queue.cc:260] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.117422 8424 ts_tablet_manager.cc:1428] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:35:53.294979 8430 raft_consensus.cc:482] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:53.295138 8430 raft_consensus.cc:504] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.296372 8430 leader_election.cc:288] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949), ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:53.300256 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.296743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44257 (local address 127.0.98.1:40949)
0927 12:35:53.297000 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.297003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.297205 (+ 202us) server_negotiation.cc:407] Connection header received
0927 12:35:53.297347 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.297350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.297390 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.297453 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:53.297691 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.298206 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.299283 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.299697 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.299790 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.299864 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.300023 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.300126 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260927 12:35:53.300268 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.296654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.98.3:44257)
0927 12:35:53.297115 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:53.297126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:53.297226 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:53.297476 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:53.297478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:53.297495 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:53.297588 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.297593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.298336 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.298339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.299148 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.299157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.299816 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.299820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.299910 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.299929 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:35:53.299995 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:35:53.300078 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":125,"threads_started":1}
I20260927 12:35:53.300805 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d506191e8d964946a6e2c45826ad0a58" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57a90c244645e9a6ad7d1533fd983c" is_pre_election: true
I20260927 12:35:53.300995 8068 raft_consensus.cc:2454] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 0.
I20260927 12:35:53.301393 8280 leader_election.cc:302] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, 6c57a90c244645e9a6ad7d1533fd983c; no voters:
I20260927 12:35:53.301697 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.296747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52647 (local address 127.0.98.2:39587)
0927 12:35:53.296890 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.296894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.297446 (+ 552us) server_negotiation.cc:407] Connection header received
0927 12:35:53.297603 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.297605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.297646 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.297707 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:35:53.298303 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.298775 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.300460 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.300993 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.301095 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.301157 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.301481 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.301570 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:35:53.301699 8430 raft_consensus.cc:2781] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:53.301749 8430 raft_consensus.cc:482] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:53.301783 8430 raft_consensus.cc:3037] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:53.302151 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.296654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39587 (local address 127.0.98.3:52647)
0927 12:35:53.297336 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:53.297347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:53.297493 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:53.297732 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:53.297735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:53.297751 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:35:53.298185 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.298192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.298901 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.298904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.300252 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.300261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.301235 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.301239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.301329 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.301362 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 12:35:53.301903 (+ 541us) client_negotiation.cc:240] Negotiation successful
0927 12:35:53.301999 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":50,"thread_start_us":56,"threads_started":1}
I20260927 12:35:53.302597 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d506191e8d964946a6e2c45826ad0a58" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491" is_pre_election: true
I20260927 12:35:53.302762 8208 raft_consensus.cc:2454] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 0.
I20260927 12:35:53.303045 8430 raft_consensus.cc:504] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.303393 8430 leader_election.cc:288] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949), ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:53.303813 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d506191e8d964946a6e2c45826ad0a58" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491"
I20260927 12:35:53.303800 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d506191e8d964946a6e2c45826ad0a58" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57a90c244645e9a6ad7d1533fd983c"
I20260927 12:35:53.303946 8068 raft_consensus.cc:3037] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:53.303989 8208 raft_consensus.cc:3037] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:53.305215 8068 raft_consensus.cc:2454] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 1.
I20260927 12:35:53.305215 8208 raft_consensus.cc:2454] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 1.
I20260927 12:35:53.305627 8282 leader_election.cc:302] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:53.305824 8430 raft_consensus.cc:2781] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:53.306025 8430 raft_consensus.cc:686] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Running, Role: LEADER
I20260927 12:35:53.306305 8430 consensus_queue.cc:237] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.308985 7909 catalog_manager.cc:5495] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ca452e86239484bac1556d74fe334e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0ca452e86239484bac1556d74fe334e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } health_report { overall_health: HEALTHY } } }
I20260927 12:35:53.356182 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.352607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.98.3:44047)
0927 12:35:53.352873 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.352880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.352893 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:53.352998 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.353000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.353055 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.353157 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:53.353337 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.353871 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.354661 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.354893 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.355002 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.355222 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.355692 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.355778 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.355909 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.355978 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":111,"threads_started":1}
I20260927 12:35:53.359223 8208 raft_consensus.cc:1261] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Refusing update from remote peer 0ca452e86239484bac1556d74fe334e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:53.359223 8068 raft_consensus.cc:1261] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Refusing update from remote peer 0ca452e86239484bac1556d74fe334e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:53.359751 8497 consensus_queue.cc:1035] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:53.360011 8430 consensus_queue.cc:1035] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:53.377936 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:53.378129 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:53.380352 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Sending DeleteTablet for 3 replicas of tablet d506191e8d964946a6e2c45826ad0a58
I20260927 12:35:53.380986 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet d506191e8d964946a6e2c45826ad0a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260927 12:35:53.381158 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet d506191e8d964946a6e2c45826ad0a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53436
I20260927 12:35:53.381223 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet d506191e8d964946a6e2c45826ad0a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260927 12:35:53.381356 8448 tablet_replica.cc:330] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: stopping tablet replica
I20260927 12:35:53.381397 8447 tablet_replica.cc:330] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: stopping tablet replica
I20260927 12:35:53.381477 8448 raft_consensus.cc:2229] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:53.381516 8447 raft_consensus.cc:2229] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:53.381642 8447 raft_consensus.cc:2258] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.381814 8446 tablet_replica.cc:330] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: stopping tablet replica
I20260927 12:35:53.381809 8448 raft_consensus.cc:2258] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.381924 8446 raft_consensus.cc:2229] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:53.382050 8446 raft_consensus.cc:2258] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.382814 8447 ts_tablet_manager.cc:1905] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.383065 8446 ts_tablet_manager.cc:1905] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.385264 8446 ts_tablet_manager.cc:1918] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:53.385334 8446 log.cc:1196] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/d506191e8d964946a6e2c45826ad0a58
I20260927 12:35:53.385581 8446 ts_tablet_manager.cc:1939] T d506191e8d964946a6e2c45826ad0a58 P ea8388b5de5b4350a87d2f4aec27c491: Deleting consensus metadata
I20260927 12:35:53.385710 8447 ts_tablet_manager.cc:1918] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:53.385803 8447 log.cc:1196] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/d506191e8d964946a6e2c45826ad0a58
I20260927 12:35:53.386166 8447 ts_tablet_manager.cc:1939] T d506191e8d964946a6e2c45826ad0a58 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting consensus metadata
I20260927 12:35:53.386436 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:35:53.386955 7898 catalog_manager.cc:4841] TS ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587): tablet d506191e8d964946a6e2c45826ad0a58 (table test-table [id=d324a881585040699f0e5ae95c101976]) successfully deleted
W20260927 12:35:53.387094 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:35:53.387226 7896 catalog_manager.cc:4841] TS 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949): tablet d506191e8d964946a6e2c45826ad0a58 (table test-table [id=d324a881585040699f0e5ae95c101976]) successfully deleted
I20260927 12:35:53.388291 8448 ts_tablet_manager.cc:1905] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.390820 8448 ts_tablet_manager.cc:1918] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:35:53.390905 8448 log.cc:1196] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/d506191e8d964946a6e2c45826ad0a58
I20260927 12:35:53.391139 8448 ts_tablet_manager.cc:1939] T d506191e8d964946a6e2c45826ad0a58 P 0ca452e86239484bac1556d74fe334e6: Deleting consensus metadata
I20260927 12:35:53.392159 7898 catalog_manager.cc:4841] TS 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047): tablet d506191e8d964946a6e2c45826ad0a58 (table test-table [id=d324a881585040699f0e5ae95c101976]) successfully deleted
I20260927 12:35:53.392547 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 028cf7d7a6004bc4b5ec436edcd0f338 (DEFAULT_TABLE table=test-table [id=3523f9521dff4c63ae4d0be3723d2501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:53.392829 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028cf7d7a6004bc4b5ec436edcd0f338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:53.392889 8328 tablet_service.cc:1467] Processing CreateTablet for tablet 028cf7d7a6004bc4b5ec436edcd0f338 (DEFAULT_TABLE table=test-table [id=3523f9521dff4c63ae4d0be3723d2501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:53.393100 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028cf7d7a6004bc4b5ec436edcd0f338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:53.395211 8422 tablet_bootstrap.cc:492] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: Bootstrap starting.
I20260927 12:35:53.395409 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 028cf7d7a6004bc4b5ec436edcd0f338 (DEFAULT_TABLE table=test-table [id=3523f9521dff4c63ae4d0be3723d2501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:35:53.395701 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028cf7d7a6004bc4b5ec436edcd0f338. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:53.396152 8422 tablet_bootstrap.cc:654] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:53.396265 8423 tablet_bootstrap.cc:492] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Bootstrap starting.
I20260927 12:35:53.397276 8423 tablet_bootstrap.cc:654] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:53.398681 8423 tablet_bootstrap.cc:492] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: No bootstrap required, opened a new log
I20260927 12:35:53.398792 8423 ts_tablet_manager.cc:1397] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:53.399267 8423 raft_consensus.cc:348] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.399384 8423 raft_consensus.cc:374] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:53.399415 8423 raft_consensus.cc:729] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Initialized, Role: FOLLOWER
I20260927 12:35:53.399665 8424 tablet_bootstrap.cc:492] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Bootstrap starting.
I20260927 12:35:53.399875 8422 tablet_bootstrap.cc:492] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: No bootstrap required, opened a new log
I20260927 12:35:53.399956 8422 ts_tablet_manager.cc:1397] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 12:35:53.400462 8424 tablet_bootstrap.cc:654] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:53.400420 8422 raft_consensus.cc:348] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.400538 8422 raft_consensus.cc:374] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:53.400568 8422 raft_consensus.cc:729] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57a90c244645e9a6ad7d1533fd983c, State: Initialized, Role: FOLLOWER
I20260927 12:35:53.400692 8422 consensus_queue.cc:260] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.400964 8422 ts_tablet_manager.cc:1428] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:35:53.401347 8423 consensus_queue.cc:260] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.401629 8423 ts_tablet_manager.cc:1428] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:35:53.402539 8424 tablet_bootstrap.cc:492] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: No bootstrap required, opened a new log
I20260927 12:35:53.402630 8424 ts_tablet_manager.cc:1397] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:53.403019 8424 raft_consensus.cc:348] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.403134 8424 raft_consensus.cc:374] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:53.403162 8424 raft_consensus.cc:729] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8388b5de5b4350a87d2f4aec27c491, State: Initialized, Role: FOLLOWER
I20260927 12:35:53.403281 8424 consensus_queue.cc:260] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.403541 8424 ts_tablet_manager.cc:1428] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:35:53.460311 8497 raft_consensus.cc:482] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:35:53.460451 8497 raft_consensus.cc:504] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.460906 8497 leader_election.cc:288] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949), ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:53.461386 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028cf7d7a6004bc4b5ec436edcd0f338" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57a90c244645e9a6ad7d1533fd983c" is_pre_election: true
I20260927 12:35:53.461370 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028cf7d7a6004bc4b5ec436edcd0f338" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491" is_pre_election: true
I20260927 12:35:53.461527 8068 raft_consensus.cc:2454] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 0.
I20260927 12:35:53.461534 8208 raft_consensus.cc:2454] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 0.
I20260927 12:35:53.461833 8282 leader_election.cc:302] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:53.462010 8497 raft_consensus.cc:2781] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:35:53.462085 8497 raft_consensus.cc:482] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:35:53.462129 8497 raft_consensus.cc:3037] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:53.463394 8497 raft_consensus.cc:504] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.463794 8497 leader_election.cc:288] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949), ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587)
I20260927 12:35:53.464237 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028cf7d7a6004bc4b5ec436edcd0f338" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57a90c244645e9a6ad7d1533fd983c"
I20260927 12:35:53.464243 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028cf7d7a6004bc4b5ec436edcd0f338" candidate_uuid: "0ca452e86239484bac1556d74fe334e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8388b5de5b4350a87d2f4aec27c491"
I20260927 12:35:53.464371 8068 raft_consensus.cc:3037] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:53.464372 8208 raft_consensus.cc:3037] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:53.465524 8068 raft_consensus.cc:2454] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 1.
I20260927 12:35:53.465559 8208 raft_consensus.cc:2454] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ca452e86239484bac1556d74fe334e6 in term 1.
I20260927 12:35:53.465867 8282 leader_election.cc:302] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ca452e86239484bac1556d74fe334e6, ea8388b5de5b4350a87d2f4aec27c491; no voters:
I20260927 12:35:53.466104 8497 raft_consensus.cc:2781] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:53.466182 8497 raft_consensus.cc:686] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ca452e86239484bac1556d74fe334e6, State: Running, Role: LEADER
I20260927 12:35:53.466316 8497 consensus_queue.cc:237] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } }
I20260927 12:35:53.467924 7909 catalog_manager.cc:5495] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ca452e86239484bac1556d74fe334e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0ca452e86239484bac1556d74fe334e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ca452e86239484bac1556d74fe334e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } health_report { overall_health: HEALTHY } } }
I20260927 12:35:53.501770 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.497749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.98.3:55809)
0927 12:35:53.498035 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:53.498047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:53.498144 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:53.498393 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:53.498395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:53.498400 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:53.498525 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.498529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.499544 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.499547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.500285 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.500296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.500925 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.500929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.501005 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.501053 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.501439 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.501446 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:35:53.501514 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:35:53.501604 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260927 12:35:53.501751 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.497889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55809 (local address 127.0.98.1:40949)
0927 12:35:53.498017 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.498022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.498115 (+ 93us) server_negotiation.cc:407] Connection header received
0927 12:35:53.498238 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.498241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.498299 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.498385 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:53.498623 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.499403 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.500405 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.500765 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.500891 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.501191 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.501290 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.501387 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.501543 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.501604 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:35:53.508529 8208 raft_consensus.cc:1261] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Refusing update from remote peer 0ca452e86239484bac1556d74fe334e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:53.508543 8068 raft_consensus.cc:1261] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Refusing update from remote peer 0ca452e86239484bac1556d74fe334e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:35:53.508908 8497 consensus_queue.cc:1035] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8388b5de5b4350a87d2f4aec27c491" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:53.509084 8503 consensus_queue.cc:1035] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c57a90c244645e9a6ad7d1533fd983c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:35:53.529696 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.525642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44047 (local address 127.0.98.1:47393)
0927 12:35:53.526001 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:53.526017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:53.526119 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:53.526350 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:53.526352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:53.526358 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:35:53.526457 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.526462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.527439 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.527442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.528281 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:53.528292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.529006 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.529008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:53.529077 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.529105 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.529390 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.529397 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:35:53.529473 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:35:53.529566 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":65,"threads_started":1}
I20260927 12:35:53.529685 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.525692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47393 (local address 127.0.98.3:44047)
0927 12:35:53.525889 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:53.525894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:53.526097 (+ 203us) server_negotiation.cc:407] Connection header received
0927 12:35:53.526203 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:53.526207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:53.526262 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:53.526341 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:35:53.526555 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.527306 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.528439 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:53.528779 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:53.528899 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:53.529228 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:35:53.529319 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:35:53.529386 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:53.529486 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 12:35:53.529542 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260927 12:35:53.556138 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:53.556326 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:35:53.558694 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 960b1329d24c444c9ca6d8f8041ea2cb: Sending DeleteTablet for 3 replicas of tablet 028cf7d7a6004bc4b5ec436edcd0f338
I20260927 12:35:53.559423 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 028cf7d7a6004bc4b5ec436edcd0f338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260927 12:35:53.559487 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet 028cf7d7a6004bc4b5ec436edcd0f338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53436
I20260927 12:35:53.559500 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 028cf7d7a6004bc4b5ec436edcd0f338 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:35:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260927 12:35:53.559648 8448 tablet_replica.cc:330] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: stopping tablet replica
I20260927 12:35:53.559641 8447 tablet_replica.cc:330] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: stopping tablet replica
I20260927 12:35:53.559677 8446 tablet_replica.cc:330] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: stopping tablet replica
I20260927 12:35:53.559767 8447 raft_consensus.cc:2229] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:53.559770 8448 raft_consensus.cc:2229] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:35:53.559777 8446 raft_consensus.cc:2229] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:35:53.559880 8446 raft_consensus.cc:2258] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.559890 8447 raft_consensus.cc:2258] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.559984 8448 raft_consensus.cc:2258] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:35:53.560664 8446 ts_tablet_manager.cc:1905] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.560784 8448 ts_tablet_manager.cc:1905] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.561496 8447 ts_tablet_manager.cc:1905] T 028cf7d7a6004bc4b5ec436edcd0f338 P 6c57a90c244645e9a6ad7d1533fd983c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:35:53.562562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 7983
I20260927 12:35:53.563777 8448 ts_tablet_manager.cc:1918] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:53.563855 8448 log.cc:1196] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/028cf7d7a6004bc4b5ec436edcd0f338
I20260927 12:35:53.564069 8446 ts_tablet_manager.cc:1918] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:35:53.564145 8448 ts_tablet_manager.cc:1939] T 028cf7d7a6004bc4b5ec436edcd0f338 P 0ca452e86239484bac1556d74fe334e6: Deleting consensus metadata
I20260927 12:35:53.564165 8446 log.cc:1196] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/028cf7d7a6004bc4b5ec436edcd0f338
I20260927 12:35:53.564404 8446 ts_tablet_manager.cc:1939] T 028cf7d7a6004bc4b5ec436edcd0f338 P ea8388b5de5b4350a87d2f4aec27c491: Deleting consensus metadata
I20260927 12:35:53.564954 7898 catalog_manager.cc:4841] TS 0ca452e86239484bac1556d74fe334e6 (127.0.98.3:44047): tablet 028cf7d7a6004bc4b5ec436edcd0f338 (table test-table [id=3523f9521dff4c63ae4d0be3723d2501]) successfully deleted
I20260927 12:35:53.565773 7898 catalog_manager.cc:4841] TS ea8388b5de5b4350a87d2f4aec27c491 (127.0.98.2:39587): tablet 028cf7d7a6004bc4b5ec436edcd0f338 (table test-table [id=3523f9521dff4c63ae4d0be3723d2501]) successfully deleted
W20260927 12:35:53.570092 7896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40949 (error 108)
I20260927 12:35:53.570155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 8123
I20260927 12:35:53.573062 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:53.572544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.0.1:55784)
0927 12:35:53.572859 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:53.572941 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":60,"threads_started":1}
W20260927 12:35:53.573246 7896 catalog_manager.cc:4584] TS 6c57a90c244645e9a6ad7d1533fd983c (127.0.98.1:40949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 028cf7d7a6004bc4b5ec436edcd0f338: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40949: connect: Connection refused (error 111)
I20260927 12:35:53.576927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 8263
I20260927 12:35:53.583189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 7879
2026-09-27T12:35:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3222 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[8521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[8521](info): set up 2 sockets
Sep 27 12:35:53 dist-test-slave-tzgx krb5kdc[8521](info): commencing operation
krb5kdc: starting...
W20260927 12:35:54.556650 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:54 dist-test-slave-tzgx krb5kdc[8521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:54.691203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40045
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:54.806293 8534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:54.806661 8534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:54.806720 8534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:54.811430 8534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:54.811528 8534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:54.811548 8534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:54.811564 8534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:54.811579 8534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:54.817512 8534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:54.818646 8534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:54.819687 8534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:54.828672 8542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:54.828740 8539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:54.828850 8534 server_base.cc:1034] running on GCE node
W20260927 12:35:54.828752 8540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:54.829646 8534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:54.830874 8534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:54.832078 8534 hybrid_clock.cc:648] HybridClock initialized: now 1790512554832053 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T12:35:54Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1253 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:35:54 dist-test-slave-tzgx krb5kdc[8547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:35:54 dist-test-slave-tzgx krb5kdc[8547](info): set up 2 sockets
Sep 27 12:35:54 dist-test-slave-tzgx krb5kdc[8547](info): commencing operation
krb5kdc: starting...
W20260927 12:35:55.804533 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:35:55 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:35:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:35:55.927404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:56.038501 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:56.038798 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:56.038841 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:56.043035 8560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:56.043107 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:56.043125 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:56.043141 8560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:56.043155 8560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:56.049261 8560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:56.050341 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:56.051306 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:35:56.060240 8560 server_base.cc:1034] running on GCE node
W20260927 12:35:56.060196 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.060135 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.060174 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:56.061004 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:56.062168 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:56.063356 8560 hybrid_clock.cc:648] HybridClock initialized: now 1790512556063336 us; error 31 us; skew 500 ppm
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:56.066461 8560 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:56.067484 8560 webserver.cc:460] Webserver started at http://127.0.98.62:44123/ using document root <none> and password file <none>
I20260927 12:35:56.068142 8560 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:56.068224 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:56.068512 8560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:56.070956 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "4f8368dc6d2840fd9a4bab2f4558ff27"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "080d65926c3488c2c7ea1754ef6dc0a8"
server_key_iv: "c60d69bf7cbc6422a34adf3cbfc1723e"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.071574 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f8368dc6d2840fd9a4bab2f4558ff27"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "080d65926c3488c2c7ea1754ef6dc0a8"
server_key_iv: "c60d69bf7cbc6422a34adf3cbfc1723e"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.076066 8560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 12:35:56.078752 8575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.080173 8560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 12:35:56.080307 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "4f8368dc6d2840fd9a4bab2f4558ff27"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "080d65926c3488c2c7ea1754ef6dc0a8"
server_key_iv: "c60d69bf7cbc6422a34adf3cbfc1723e"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.080423 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:56.093021 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:56.096534 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:56.096732 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:56.103549 8560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35657
I20260927 12:35:56.103605 8627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35657 every 8 connection(s)
I20260927 12:35:56.104604 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:56.107542 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:35:56.113422 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Bootstrap starting.
I20260927 12:35:56.113937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 8560
I20260927 12:35:56.114063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:35:56.114326 392 external_mini_cluster.cc:1428] Setting key 22274fb8461ea2e8edc03d7ec547ea82
I20260927 12:35:56.116178 8628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Neither blocks nor log segments found. Creating new log.
I20260927 12:35:56.116925 8628 log.cc:824] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:56.118800 8628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: No bootstrap required, opened a new log
I20260927 12:35:56.122352 8628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:56.122612 8628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:35:56.122663 8628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8368dc6d2840fd9a4bab2f4558ff27, State: Initialized, Role: FOLLOWER
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:56.123138 8628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:56.123279 8628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:56.123340 8628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:56.123427 8628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:35:56.124385 8628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:56.124778 8628 leader_election.cc:302] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f8368dc6d2840fd9a4bab2f4558ff27; no voters:
I20260927 12:35:56.125124 8628 leader_election.cc:288] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:35:56.125241 8633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:35:56.125404 8633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8368dc6d2840fd9a4bab2f4558ff27, State: Running, Role: LEADER
I20260927 12:35:56.125720 8633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:56.126478 8628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:56.128127 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f8368dc6d2840fd9a4bab2f4558ff27. Latest consensus state: current_term: 1 leader_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } } }
I20260927 12:35:56.128279 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:56.128834 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.115657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.0.98.62:35657)
0927 12:35:56.116123 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:56.116146 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:56.116184 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:35:56.116877 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:56.116895 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:56.117271 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:56.117603 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:56.118389 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.119425 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.120315 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.120809 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.120972 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.123395 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:56.123403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:56.123423 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:56.123459 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:56.125440 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.125934 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.125940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.125946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.126083 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.126356 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.126359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.126361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.126840 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:56.127047 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:56.127347 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 12:35:56.127691 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260927 12:35:56.129629 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } } }
I20260927 12:35:56.129787 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: This master's current role is: LEADER
W20260927 12:35:56.131436 8647 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:35:56.131517 8647 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:35:56.131783 8648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:56.132759 8648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:56.138548 8648 catalog_manager.cc:1347] Generated new cluster ID: 5db373bd02824f03b3667d7859beb3dc
I20260927 12:35:56.138731 8648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:56.155448 8648 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:35:56.156798 8648 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:56.162185 8648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Generated new TSK 0
I20260927 12:35:56.162912 8648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:35:56.198859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35657
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:56.313910 8654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:56.314222 8654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:56.314267 8654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:56.318466 8654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:56.318545 8654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:56.318607 8654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:56.324035 8654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:56.325217 8654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:56.326287 8654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:56.336239 8659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.336225 8660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:56.336386 8654 server_base.cc:1034] running on GCE node
W20260927 12:35:56.336627 8662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:56.337037 8654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:56.337703 8654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:56.338877 8654 hybrid_clock.cc:648] HybridClock initialized: now 1790512556338856 us; error 37 us; skew 500 ppm
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:56.341799 8654 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:35:56.342720 8654 webserver.cc:460] Webserver started at http://127.0.98.1:34521/ using document root <none> and password file <none>
I20260927 12:35:56.343261 8654 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:56.343320 8654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:56.343523 8654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:56.345669 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "758e9ce818464ece86b36ffe9fdc15e3"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "6b99b8e2a2fe88a57ada28195524c654"
server_key_iv: "f81dab3e9579d5382fb46b3024cf4e95"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.346202 8654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "758e9ce818464ece86b36ffe9fdc15e3"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "6b99b8e2a2fe88a57ada28195524c654"
server_key_iv: "f81dab3e9579d5382fb46b3024cf4e95"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.350220 8654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:56.353001 8669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.354282 8654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:56.354408 8654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "758e9ce818464ece86b36ffe9fdc15e3"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "6b99b8e2a2fe88a57ada28195524c654"
server_key_iv: "f81dab3e9579d5382fb46b3024cf4e95"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.354535 8654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:56.367821 8654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:56.371246 8654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:56.371459 8654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:56.372148 8654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:56.373124 8654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:56.373184 8654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.373239 8654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:56.373277 8654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.382623 8654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44665
I20260927 12:35:56.382655 8782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44665 every 8 connection(s)
I20260927 12:35:56.383589 8654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:56.385317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 8654
I20260927 12:35:56.385440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:35:56.385752 392 external_mini_cluster.cc:1428] Setting key 41b392c888d4a28f50f002337f0eec7e
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:35:56.400038 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.386190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55133 (local address 127.0.98.62:35657)
0927 12:35:56.386344 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:56.386350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:56.386787 (+ 437us) server_negotiation.cc:407] Connection header received
0927 12:35:56.388278 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:56.388282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:56.388342 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:56.388436 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:56.390087 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.390969 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.392318 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.392697 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.392838 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.395693 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:56.395697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:56.395699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:56.395727 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:56.397416 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.398229 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.398232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.398235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.398324 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.398700 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.398703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.398704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.398917 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:56.399008 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:56.399711 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 12:35:56.399851 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:35:56.401019 8785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.386120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:55133)
0927 12:35:56.386626 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:56.386683 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:56.388021 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:56.388662 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:56.388672 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:56.389182 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:56.389824 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:56.389845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.391127 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.391130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:56.392182 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:56.392194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.392901 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.392905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:56.393088 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.393705 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:56.393721 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:56.395406 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:56.397697 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:56.397712 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:56.397728 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:56.398061 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:56.398420 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:56.398422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:56.398424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:56.398587 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:56.399036 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:56.399044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:56.399406 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 12:35:56.399624 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 12:35:56.400036 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260927 12:35:56.402453 8783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35657
I20260927 12:35:56.402710 8783 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:56.403245 8783 heartbeater.cc:511] Master 127.0.98.62:35657 requested a full tablet report, sending...
I20260927 12:35:56.404866 8592 ts_manager.cc:194] Registered new tserver with Master: 758e9ce818464ece86b36ffe9fdc15e3 (127.0.98.1:44665)
I20260927 12:35:56.406010 8592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:35:56.415741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35657
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:56.529546 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:56.529850 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:56.529897 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:56.534274 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:56.534343 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:56.534404 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:56.540055 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:56.541296 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:56.542359 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:56.552073 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.552107 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.552102 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:56.552218 8788 server_base.cc:1034] running on GCE node
I20260927 12:35:56.552824 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:56.553439 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:56.554602 8788 hybrid_clock.cc:648] HybridClock initialized: now 1790512556554575 us; error 41 us; skew 500 ppm
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:56.557744 8788 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:35:56.558732 8788 webserver.cc:460] Webserver started at http://127.0.98.2:42915/ using document root <none> and password file <none>
I20260927 12:35:56.559284 8788 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:56.559350 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:56.559540 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:56.561951 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "4438fb8ce4b3436ead371b7aa7df4944"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "7849c9e315d1000e2b60fd7962e16d4f"
server_key_iv: "38137bffa387ea10fa6ca77cc2589927"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.562606 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4438fb8ce4b3436ead371b7aa7df4944"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "7849c9e315d1000e2b60fd7962e16d4f"
server_key_iv: "38137bffa387ea10fa6ca77cc2589927"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.566879 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:35:56.570153 8803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.572427 8788 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:35:56.572603 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "4438fb8ce4b3436ead371b7aa7df4944"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "7849c9e315d1000e2b60fd7962e16d4f"
server_key_iv: "38137bffa387ea10fa6ca77cc2589927"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.572738 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:56.593467 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:56.596762 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:56.596958 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:56.597568 8788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:56.598415 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:56.598471 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.598524 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:56.598552 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.606907 8788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43703
I20260927 12:35:56.606964 8916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43703 every 8 connection(s)
I20260927 12:35:56.607997 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:56.612411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 8788
I20260927 12:35:56.612555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:35:56.612838 392 external_mini_cluster.cc:1428] Setting key 5263e3c93ffb2a24014ad75348cb4765
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:56.629945 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.609907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38033 (local address 127.0.98.62:35657)
0927 12:35:56.610083 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:56.610087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:56.610907 (+ 820us) server_negotiation.cc:407] Connection header received
0927 12:35:56.612379 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:56.612382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:56.612430 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:56.612525 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:56.613899 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.614468 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.616184 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.616508 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.616612 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.625704 (+ 9092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:56.625709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:56.625712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:56.625749 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:56.627473 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.628241 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.628246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.628248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.628374 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.628728 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.628731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.628732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.628945 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:56.629038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:56.629628 (+ 590us) server_negotiation.cc:299] Negotiation successful
0927 12:35:56.629765 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 12:35:56.630733 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.610289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:38033)
0927 12:35:56.610756 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:56.610800 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:56.612127 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:56.612701 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:56.612711 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:56.613192 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:56.613761 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:56.613780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.614601 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.614606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:56.616010 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:56.616024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.616641 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.616644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:56.616824 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.621080 (+ 4256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:56.621100 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:56.625444 (+ 4344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:56.627638 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:56.627646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:56.627666 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:56.628113 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:56.628461 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:56.628464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:56.628465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:56.628609 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:56.629050 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:56.629058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:56.629331 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 12:35:56.629534 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 12:35:56.629828 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260927 12:35:56.632078 8917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35657
I20260927 12:35:56.632397 8917 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:56.633018 8917 heartbeater.cc:511] Master 127.0.98.62:35657 requested a full tablet report, sending...
I20260927 12:35:56.634263 8592 ts_manager.cc:194] Registered new tserver with Master: 4438fb8ce4b3436ead371b7aa7df4944 (127.0.98.2:43703)
I20260927 12:35:56.634857 8592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:35:56.645740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35657
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:56.755851 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:56.756175 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:56.756225 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:56.760488 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:56.760548 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:56.760608 8922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:56.765988 8922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:56.767023 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:56.768025 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:56.777619 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.777693 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:56.777693 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:56.777885 8922 server_base.cc:1034] running on GCE node
I20260927 12:35:56.778290 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:56.778961 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:56.780131 8922 hybrid_clock.cc:648] HybridClock initialized: now 1790512556780110 us; error 34 us; skew 500 ppm
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:56.782809 8922 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:35:56.783667 8922 webserver.cc:460] Webserver started at http://127.0.98.3:38507/ using document root <none> and password file <none>
I20260927 12:35:56.784219 8922 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:56.784279 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:56.784493 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:35:56.786478 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "7f7a52d013e946d68dc1dfc7de9d872a"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "48011afdf8da3ff7b45d731317d6afed"
server_key_iv: "e356525444c81f30690595c468f1fcd5"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.786990 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7f7a52d013e946d68dc1dfc7de9d872a"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "48011afdf8da3ff7b45d731317d6afed"
server_key_iv: "e356525444c81f30690595c468f1fcd5"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.790652 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:35:56.793144 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.794565 8922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:56.794685 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "7f7a52d013e946d68dc1dfc7de9d872a"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "48011afdf8da3ff7b45d731317d6afed"
server_key_iv: "e356525444c81f30690595c468f1fcd5"
server_key_version: "encryptionkey@0"
I20260927 12:35:56.794793 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:56.816510 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:56.819463 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:56.819648 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:56.820254 8922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:56.821096 8922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:56.821154 8922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.821208 8922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:56.821233 8922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:56.830022 8922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40265
I20260927 12:35:56.830060 9050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40265 every 8 connection(s)
I20260927 12:35:56.830911 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:56.832268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 8922
I20260927 12:35:56.832389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:35:56.832648 392 external_mini_cluster.cc:1428] Setting key 622b30d7d2f015dd9e7759393dfc85c7
Sep 27 12:35:56 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:35:56.844786 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.832854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50627 (local address 127.0.98.62:35657)
0927 12:35:56.833014 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:56.833019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:56.833944 (+ 925us) server_negotiation.cc:407] Connection header received
0927 12:35:56.835425 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:56.835428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:56.835478 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:56.835572 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:56.836841 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.837394 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.838638 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.838871 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.838961 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.841526 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:56.841530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:56.841532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:56.841557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:56.842946 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.843458 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.843460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.843461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.843547 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:56.843811 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:56.843812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:56.843813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:56.843979 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:56.844051 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:56.844563 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 12:35:56.844660 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:35:56.845846 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:56.833290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:50627)
0927 12:35:56.833790 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:56.833849 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:56.835200 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:56.835731 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:56.835738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:56.836184 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:56.836692 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:56.836706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.837541 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.837544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:56.838499 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:56.838509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:56.839010 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:56.839013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:56.839163 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:56.839769 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:56.839787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:56.841343 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:35:56.843082 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:56.843088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:56.843101 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:56.843379 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:56.843610 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:35:56.843612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:35:56.843613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:35:56.843741 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:35:56.844068 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:35:56.844073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:35:56.844369 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 12:35:56.844530 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 12:35:56.844843 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260927 12:35:56.846868 9051 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35657
I20260927 12:35:56.847095 9051 heartbeater.cc:464] Registering TS with master...
I20260927 12:35:56.847577 9051 heartbeater.cc:511] Master 127.0.98.62:35657 requested a full tablet report, sending...
I20260927 12:35:56.848637 8591 ts_manager.cc:194] Registered new tserver with Master: 7f7a52d013e946d68dc1dfc7de9d872a (127.0.98.3:40265)
I20260927 12:35:56.849255 8591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50627
I20260927 12:35:56.858753 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:35:56.934720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 8654
I20260927 12:35:56.939898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 8788
I20260927 12:35:56.944460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 8922
I20260927 12:35:56.949110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 8560
I20260927 12:35:56.954715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35657
--webserver_interface=127.0.98.62
--webserver_port=44123
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:35:57.067963 9060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:57.068351 9060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:57.068418 9060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:57.072860 9060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:35:57.072952 9060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:57.073015 9060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:57.073041 9060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:35:57.073071 9060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:35:57.078999 9060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:57.080359 9060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:57.081523 9060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:57.090771 9066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:57.090806 9065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:57.090796 9060 server_base.cc:1034] running on GCE node
W20260927 12:35:57.090775 9068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:57.091622 9060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:57.092892 9060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:57.094106 9060 hybrid_clock.cc:648] HybridClock initialized: now 1790512557094078 us; error 50 us; skew 500 ppm
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:57.097200 9060 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:35:57.098132 9060 webserver.cc:460] Webserver started at http://127.0.98.62:44123/ using document root <none> and password file <none>
I20260927 12:35:57.098683 9060 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:57.098757 9060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:57.102880 9060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:57.105399 9075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.106874 9060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:35:57.106992 9060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "4f8368dc6d2840fd9a4bab2f4558ff27"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "080d65926c3488c2c7ea1754ef6dc0a8"
server_key_iv: "c60d69bf7cbc6422a34adf3cbfc1723e"
server_key_version: "encryptionkey@0"
I20260927 12:35:57.107539 9060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:57.117744 9060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:57.121176 9060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:57.121352 9060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:57.129889 9060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35657
I20260927 12:35:57.129921 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35657 every 8 connection(s)
I20260927 12:35:57.131060 9060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:35:57.135862 9128 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:35:57.137758 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Bootstrap starting.
I20260927 12:35:57.141309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9060
I20260927 12:35:57.141757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44665
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35657
--webserver_port=34521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:35:57.154904 9128 log.cc:824] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Log is configured to *not* fsync() on all Append() calls
I20260927 12:35:57.162187 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:35:57.162547 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Bootstrap complete.
I20260927 12:35:57.166601 9128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:57.166996 9128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8368dc6d2840fd9a4bab2f4558ff27, State: Initialized, Role: FOLLOWER
I20260927 12:35:57.167471 9128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:57.167598 9128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:35:57.167665 9128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:35:57.167761 9128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:35:57.169750 9128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:57.170087 9128 leader_election.cc:302] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f8368dc6d2840fd9a4bab2f4558ff27; no voters:
I20260927 12:35:57.170408 9128 leader_election.cc:288] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:35:57.170509 9132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:35:57.170734 9132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [term 2 LEADER]: Becoming Leader. State: Replica: 4f8368dc6d2840fd9a4bab2f4558ff27, State: Running, Role: LEADER
I20260927 12:35:57.171128 9132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } }
I20260927 12:35:57.171698 9128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:35:57.172539 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } } }
I20260927 12:35:57.172665 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:57.173089 9134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f8368dc6d2840fd9a4bab2f4558ff27. Latest consensus state: current_term: 2 leader_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8368dc6d2840fd9a4bab2f4558ff27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35657 } } }
I20260927 12:35:57.173218 9134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27 [sys.catalog]: This master's current role is: LEADER
I20260927 12:35:57.173494 9141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:35:57.176880 9141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:35:57.177513 9141 catalog_manager.cc:1259] Loaded cluster ID: 5db373bd02824f03b3667d7859beb3dc
I20260927 12:35:57.177572 9141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:35:57.179853 9141 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:35:57.180539 9141 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f8368dc6d2840fd9a4bab2f4558ff27: Loaded TSK: 0
I20260927 12:35:57.182667 9141 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:35:57.257916 9130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:57.258177 9130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:57.258222 9130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:57.262481 9130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:57.262562 9130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:57.262624 9130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:35:57.268035 9130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34521
--tserver_master_addrs=127.0.98.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:57.269140 9130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:57.270152 9130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:57.280154 9155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:57.280159 9154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:57.280189 9157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:57.280591 9130 server_base.cc:1034] running on GCE node
I20260927 12:35:57.281147 9130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:57.282114 9130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:57.283346 9130 hybrid_clock.cc:648] HybridClock initialized: now 1790512557283317 us; error 43 us; skew 500 ppm
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:57.287317 9130 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:35:57.288380 9130 webserver.cc:460] Webserver started at http://127.0.98.1:34521/ using document root <none> and password file <none>
I20260927 12:35:57.288973 9130 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:57.289050 9130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:57.293351 9130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:35:57.295821 9164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.297310 9130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:35:57.297465 9130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "758e9ce818464ece86b36ffe9fdc15e3"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "6b99b8e2a2fe88a57ada28195524c654"
server_key_iv: "f81dab3e9579d5382fb46b3024cf4e95"
server_key_version: "encryptionkey@0"
I20260927 12:35:57.298094 9130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:57.317390 9130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:57.320696 9130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:57.320886 9130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:57.321529 9130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:57.322386 9130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:57.322434 9130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.322487 9130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:57.322515 9130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.331202 9130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44665
I20260927 12:35:57.331274 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44665 every 8 connection(s)
I20260927 12:35:57.332278 9130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:35:57.338526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9130
I20260927 12:35:57.338936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43703
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35657
--webserver_port=42915
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.346381 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.334717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:35787)
0927 12:35:57.335225 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.335278 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.336621 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.337774 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.337782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.338214 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.338696 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.338714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.340013 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.340016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.340960 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.340971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.341753 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.341757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.341946 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.342510 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.342527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.345312 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
W20260927 12:35:57.346374 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.334576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35787 (local address 127.0.98.62:35657)
0927 12:35:57.335101 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.335122 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.335391 (+ 269us) server_negotiation.cc:407] Connection header received
0927 12:35:57.337058 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.337068 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.337376 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.337673 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.338860 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.339880 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.341086 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.341584 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.341749 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.345608 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35787: BlockingRecv error: recv got EOF from 127.0.98.1:35787 (error 108)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":125,"threads_started":1}
W20260927 12:35:57.346938 9278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35657 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35657: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.352682 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.347410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:40651)
0927 12:35:57.347533 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.347547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.347633 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.347904 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.347906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.348065 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.348197 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.348204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.348921 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.348925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.349494 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.349501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.349926 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.349930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.350020 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.350470 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.350485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.352539 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:35:57.352808 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.347460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40651 (local address 127.0.98.62:35657)
0927 12:35:57.347564 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.347569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.347586 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:35:57.347720 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.347723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.347768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.347877 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.348299 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.348824 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.349599 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.349825 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.349923 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.352706 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40651: BlockingRecv error: recv got EOF from 127.0.98.1:40651 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.359154 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.353473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:54175)
0927 12:35:57.353659 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.353672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.353764 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.354024 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.354027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.354209 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.354309 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.354316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.355097 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.355101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.355993 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.356002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.356494 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.356498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.356585 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.357042 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.357055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.359041 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 12:35:57.359417 9278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:35:57.359634 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.353556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54175 (local address 127.0.98.62:35657)
0927 12:35:57.353667 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.353670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.353717 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:35:57.353854 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.353857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.353899 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.353975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.354441 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.354999 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.356158 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.356384 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.356471 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.359211 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54175: BlockingRecv error: recv got EOF from 127.0.98.1:54175 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 12:35:57.456362 9282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:57.456642 9282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:57.456686 9282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:57.460886 9282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:57.460974 9282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:57.461038 9282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:35:57.466566 9282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42915
--tserver_master_addrs=127.0.98.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:57.467595 9282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:57.468596 9282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:57.478152 9287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:57.478157 9288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:57.478125 9290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:57.478480 9282 server_base.cc:1034] running on GCE node
I20260927 12:35:57.478889 9282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:57.479481 9282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:57.480666 9282 hybrid_clock.cc:648] HybridClock initialized: now 1790512557480645 us; error 31 us; skew 500 ppm
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:57.483662 9282 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:35:57.484668 9282 webserver.cc:460] Webserver started at http://127.0.98.2:42915/ using document root <none> and password file <none>
I20260927 12:35:57.485225 9282 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:57.485306 9282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:57.489265 9282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 12:35:57.491430 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.492798 9282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:35:57.492918 9282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "4438fb8ce4b3436ead371b7aa7df4944"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "7849c9e315d1000e2b60fd7962e16d4f"
server_key_iv: "38137bffa387ea10fa6ca77cc2589927"
server_key_version: "encryptionkey@0"
I20260927 12:35:57.493474 9282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:57.511819 9282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:57.514788 9282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:57.514966 9282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:57.515554 9282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:57.516409 9282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:57.516456 9282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.516515 9282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:57.516547 9282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.525199 9282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43703
I20260927 12:35:57.525238 9410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43703 every 8 connection(s)
I20260927 12:35:57.526219 9282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:35:57.535461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9282
I20260927 12:35:57.536005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40265
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35657
--webserver_port=38507
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:35:57.538498 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.528048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53375 (local address 127.0.98.62:35657)
0927 12:35:57.528222 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.528227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.529159 (+ 932us) server_negotiation.cc:407] Connection header received
0927 12:35:57.530578 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.530581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.530634 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.530715 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.531924 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.532551 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.533967 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.534223 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.534313 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.538358 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53375: BlockingRecv error: recv got EOF from 127.0.98.2:53375 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 12:35:57.539264 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.528468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:53375)
0927 12:35:57.528994 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.529040 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.530336 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.530880 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.530887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.531246 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.531755 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.531776 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.532716 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.532720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.533823 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.533835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.534400 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.534403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.534563 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.535369 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.535387 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.538181 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":130,"threads_started":1}
W20260927 12:35:57.539886 9411 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35657 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35657: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.546303 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.540391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:55089)
0927 12:35:57.540519 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.540533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.540610 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.540852 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.540854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.541004 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.541105 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.541110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.542193 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.542197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.542782 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.542792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.543399 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.543403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.543486 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.544127 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.544144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.546190 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 12:35:57.546551 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.540529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55089 (local address 127.0.98.62:35657)
0927 12:35:57.540641 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.540645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.540659 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:35:57.540707 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.540709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.540753 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.540834 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.541240 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.542029 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.542910 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.543258 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.543388 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.546410 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55089: BlockingRecv error: recv got EOF from 127.0.98.2:55089 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.552490 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.546863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:42951)
0927 12:35:57.546970 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.546984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.547165 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.547458 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.547461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.547598 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.547693 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.547698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.548804 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.548808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.549392 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.549402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.549971 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.549974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.550038 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.550529 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.550540 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.552384 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 12:35:57.552650 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.546986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42951 (local address 127.0.98.62:35657)
0927 12:35:57.547113 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.547117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.547133 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:35:57.547260 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.547264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.547314 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.547412 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.547809 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.548673 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.549508 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.549857 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.549959 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.552534 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42951: BlockingRecv error: recv got EOF from 127.0.98.2:42951 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 12:35:57.552722 9411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:35:57.651459 9414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:35:57.651757 9414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:35:57.651808 9414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:35:57.656028 9414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:35:57.656129 9414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:35:57.656198 9414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:35:57.661715 9414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38507
--tserver_master_addrs=127.0.98.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:35:57.662806 9414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:35:57.663805 9414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:35:57.673481 9422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:35:57.673485 9420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:57.673691 9414 server_base.cc:1034] running on GCE node
W20260927 12:35:57.673898 9419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:35:57.674327 9414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:35:57.674966 9414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:35:57.676153 9414 hybrid_clock.cc:648] HybridClock initialized: now 1790512557676128 us; error 33 us; skew 500 ppm
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512557, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:35:57.678934 9414 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:35:57.679952 9414 webserver.cc:460] Webserver started at http://127.0.98.3:38507/ using document root <none> and password file <none>
I20260927 12:35:57.680547 9414 fs_manager.cc:359] Metadata directory not provided
I20260927 12:35:57.680629 9414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:35:57.684700 9414 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:35:57.686854 9429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.688258 9414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:35:57.688364 9414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "7f7a52d013e946d68dc1dfc7de9d872a"
format_stamp: "Formatted at 2026-09-27 12:35:56 on dist-test-slave-tzgx"
server_key: "48011afdf8da3ff7b45d731317d6afed"
server_key_iv: "e356525444c81f30690595c468f1fcd5"
server_key_version: "encryptionkey@0"
I20260927 12:35:57.688903 9414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:35:57.705812 9414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:35:57.709199 9414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:35:57.709383 9414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:35:57.710007 9414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:35:57.710887 9414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:35:57.710935 9414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.710978 9414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:35:57.711014 9414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:35:57.719882 9414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40265
I20260927 12:35:57.719937 9542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40265 every 8 connection(s)
I20260927 12:35:57.721030 9414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:35:57.723246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9414
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:35:57.732841 9280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.722887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55879 (local address 127.0.98.62:35657)
0927 12:35:57.723051 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.723056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.724005 (+ 949us) server_negotiation.cc:407] Connection header received
0927 12:35:57.725510 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.725513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.725560 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.725635 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.726793 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.727363 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.728652 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.728911 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.728982 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.732718 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55879: BlockingRecv error: recv got EOF from 127.0.98.3:55879 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:35:57.733006 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.724285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.98.62:35657)
0927 12:35:57.724509 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.724512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.724523 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:35:57.724558 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.724561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.724597 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.724674 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.725441 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.726052 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.726934 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.727173 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.727251 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.728790 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:35:57.728796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:35:57.728811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:35:57.728842 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:35:57.730718 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:57.731147 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:57.731153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:57.731160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:57.731320 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:35:57.731633 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:35:57.731636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:35:57.731637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:35:57.732058 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:35:57.732270 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 12:35:57.732562 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 12:35:57.732819 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260927 12:35:57.733646 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.723330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:55879)
0927 12:35:57.723838 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.723898 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.725317 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.725788 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.725795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.726152 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.726654 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.726667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.727498 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.727501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.728528 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.728539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.729063 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.729067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.729240 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.729755 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.729776 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.732589 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":127,"threads_started":1}
W20260927 12:35:57.734233 9543 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35657 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35657: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.740499 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.734668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:41153)
0927 12:35:57.734768 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.734782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.734861 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.735129 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.735132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.735310 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.735463 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.735471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.736520 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.736522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.737144 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.737155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.737729 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.737732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.737812 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.738482 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.738497 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.740377 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 12:35:57.740674 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.734798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41153 (local address 127.0.98.62:35657)
0927 12:35:57.734918 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.734922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.734934 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:35:57.734965 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.734968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.735011 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.735080 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.735601 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.736398 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.737285 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.737598 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.737687 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.740543 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41153: BlockingRecv error: recv got EOF from 127.0.98.3:41153 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:57 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:57.746511 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:57.741120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:35337)
0927 12:35:57.741227 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:57.741241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:57.741316 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:57.741612 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:57.741614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:57.741749 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:57.741841 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.741846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.742655 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.742658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.743214 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:57.743223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.743675 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.743678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:57.743737 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.744246 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:57.744259 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:57.746387 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 12:35:57.746724 9543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:35:57.746927 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:57.741219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35337 (local address 127.0.98.62:35657)
0927 12:35:57.741404 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:57.741408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:57.741421 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:35:57.741461 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:57.741463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:57.741498 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:57.741574 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:57.741970 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.742539 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.743340 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:57.743580 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:57.743673 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:57.746589 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35337: BlockingRecv error: recv got EOF from 127.0.98.3:35337 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 12:35:58 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:58 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:58.365249 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:58.360133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:49735)
0927 12:35:58.360398 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:58.360411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:58.360503 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:58.360733 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:58.360735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:58.360924 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:58.361044 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:58.361050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.361762 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.361765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:58.362387 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:58.362396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.362839 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.362841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:58.362901 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:58.363408 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:58.363419 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:58.365120 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260927 12:35:58.365447 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:58.360223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49735 (local address 127.0.98.62:35657)
0927 12:35:58.360473 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:58.360477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:58.360491 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:35:58.360587 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:58.360590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:58.360632 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:58.360710 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:58.361139 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.361680 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.362514 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.362743 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.362841 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:58.365305 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49735: BlockingRecv error: recv got EOF from 127.0.98.1:49735 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
Sep 27 12:35:58 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:58 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:58.559456 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:58.553555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:52899)
0927 12:35:58.553913 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:58.553936 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:58.554136 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:58.554495 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:58.554498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:58.554695 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:58.554811 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:58.554817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.555593 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.555596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:58.556267 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:58.556278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.556726 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.556730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:58.556833 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:58.557360 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:58.557372 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:58.559324 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":89,"threads_started":1}
W20260927 12:35:58.559643 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:58.553721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52899 (local address 127.0.98.62:35657)
0927 12:35:58.553902 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:58.553906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:58.554018 (+ 112us) server_negotiation.cc:407] Connection header received
0927 12:35:58.554318 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:58.554321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:58.554375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:58.554449 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:58.554935 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.555468 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.556383 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.556615 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.556705 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:58.559513 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52899: BlockingRecv error: recv got EOF from 127.0.98.2:52899 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 12:35:58 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:58 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:58.754386 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:58.747456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:50191)
0927 12:35:58.747778 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:58.747795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:58.747916 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:58.748217 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:58.748221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:58.748411 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:58.748534 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:58.748541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.749308 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.749312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:58.750120 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:58.750148 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.750800 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.750803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:58.750893 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:58.751557 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:58.751572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:58.754200 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
W20260927 12:35:58.754513 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:58.747645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50191 (local address 127.0.98.62:35657)
0927 12:35:58.747814 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:58.747820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:58.747856 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:35:58.748034 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:58.748037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:58.748081 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:58.748173 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:58.748658 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.749200 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.750389 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:58.750655 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:58.750769 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:58.754405 (+ 3636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50191: BlockingRecv error: recv got EOF from 127.0.98.3:50191 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 12:35:59 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:59 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:59.373309 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:59.366130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:54843)
0927 12:35:59.366377 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:59.366391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:59.366481 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:59.366764 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:59.366800 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:59.367011 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:59.367129 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:59.367135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.368219 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.368222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:59.368799 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:59.368808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.369485 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.369491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:59.369637 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:59.370564 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:59.370586 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:59.373135 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
W20260927 12:35:59.373438 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:59.366248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54843 (local address 127.0.98.62:35657)
0927 12:35:59.366501 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:59.366504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:59.366545 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:35:59.366587 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:59.366589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:59.366631 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:59.366707 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:59.367269 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.367976 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.368990 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.369246 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.369376 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:59.373329 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54843: BlockingRecv error: recv got EOF from 127.0.98.1:54843 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
Sep 27 12:35:59 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:59 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:59.566188 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:59.560452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:49025)
0927 12:35:59.560778 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:59.560793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:59.560918 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:59.561176 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:59.561179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:59.561410 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:59.561509 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:59.561515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.562255 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.562259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:59.562815 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:59.562823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.563241 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.563245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:59.563324 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:59.563842 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:59.563855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:59.566044 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":86,"threads_started":1}
W20260927 12:35:59.566357 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:59.560576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49025 (local address 127.0.98.62:35657)
0927 12:35:59.560754 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:59.560758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:59.560868 (+ 110us) server_negotiation.cc:407] Connection header received
0927 12:35:59.561020 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:59.561023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:59.561067 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:59.561133 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:59.561627 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.562149 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.562929 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.563138 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.563224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:59.566219 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49025: BlockingRecv error: recv got EOF from 127.0.98.2:49025 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 12:35:59 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:35:59 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:35:59.761142 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:35:59.755448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:53923)
0927 12:35:59.755776 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 12:35:59.755790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:35:59.755896 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:35:59.756151 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:35:59.756154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:35:59.756326 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:35:59.756427 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:59.756433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.757222 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.757224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:59.757777 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:35:59.757783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.758253 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.758256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:35:59.758321 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:59.758856 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:35:59.758867 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:35:59.760956 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
W20260927 12:35:59.761226 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:35:59.755636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53923 (local address 127.0.98.62:35657)
0927 12:35:59.755778 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:35:59.755790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:35:59.755857 (+ 67us) server_negotiation.cc:407] Connection header received
0927 12:35:59.755972 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:35:59.755975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:35:59.756021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:35:59.756117 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:35:59.756565 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.757125 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.757906 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:35:59.758132 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:35:59.758223 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:35:59.761103 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53923: BlockingRecv error: recv got EOF from 127.0.98.3:53923 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 12:36:00 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:00 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:00.380937 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:00.374469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:37773)
0927 12:36:00.374760 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:00.374775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:00.374871 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:00.375314 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:00.375316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:00.375522 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:00.375627 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:00.375633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.376495 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.376497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:00.377148 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:00.377160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.377676 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.377679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:00.377755 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:00.378319 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:00.378333 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:00.380791 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
W20260927 12:36:00.381088 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:00.374564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37773 (local address 127.0.98.62:35657)
0927 12:36:00.374808 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:00.374813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:00.374829 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:36:00.375099 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:00.375102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:00.375163 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:00.375255 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:00.375773 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.376352 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.377310 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.377560 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.377647 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:00.380956 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37773: BlockingRecv error: recv got EOF from 127.0.98.1:37773 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
Sep 27 12:36:00 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:00 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:00.573351 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:00.567214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:45629)
0927 12:36:00.567500 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:00.567516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:00.567631 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:00.568037 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:00.568041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:00.568280 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:00.568389 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:00.568396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.569241 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.569254 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:00.569904 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:00.569914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.570391 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.570393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:00.570458 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:00.570929 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:00.570940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:00.573178 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
W20260927 12:36:00.573475 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:00.567347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45629 (local address 127.0.98.62:35657)
0927 12:36:00.567510 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:00.567514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:00.567596 (+ 82us) server_negotiation.cc:407] Connection header received
0927 12:36:00.567807 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:00.567810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:00.567859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:00.567957 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:00.568497 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.569135 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.570062 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.570283 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.570376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:00.573358 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45629: BlockingRecv error: recv got EOF from 127.0.98.2:45629 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 12:36:00 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:00 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:00.766988 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:00.761879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:40541)
0927 12:36:00.762139 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:00.762153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:00.762268 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:00.762504 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:00.762507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:00.762692 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:00.762783 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:00.762789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.763534 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.763537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:00.764081 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:00.764126 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.764546 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.764549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:00.764623 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:00.765119 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:00.765131 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:00.766816 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
W20260927 12:36:00.767081 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:00.761971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40541 (local address 127.0.98.62:35657)
0927 12:36:00.762137 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:00.762142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:00.762224 (+ 82us) server_negotiation.cc:407] Connection header received
0927 12:36:00.762352 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:00.762355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:00.762404 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:00.762472 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:00.762893 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.763431 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.764227 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:00.764450 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:00.764534 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:00.766979 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40541: BlockingRecv error: recv got EOF from 127.0.98.3:40541 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 12:36:01 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:01 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:01.387329 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:01.381823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:39587)
0927 12:36:01.382078 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:01.382092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:01.382189 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:01.382531 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:01.382533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:01.382728 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:01.382829 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:01.382835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.383625 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.383628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:01.384294 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:01.384304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.384736 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.384738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:01.384796 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:01.385326 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:01.385338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:01.387188 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
W20260927 12:36:01.387476 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:01.381987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39587 (local address 127.0.98.62:35657)
0927 12:36:01.382269 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:01.382272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:01.382286 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:36:01.382326 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:01.382329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:01.382367 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:01.382455 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:01.382969 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.383513 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.384410 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.384640 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.384721 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:01.387344 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39587: BlockingRecv error: recv got EOF from 127.0.98.1:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
Sep 27 12:36:01 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:01 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:01.580252 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:01.574249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:50403)
0927 12:36:01.574526 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:01.574542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:01.574651 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:01.574940 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:01.574943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:01.575129 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:01.575238 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:01.575246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.576169 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.576173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:01.576743 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:01.576753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.577349 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.577352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:01.577432 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:01.577969 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:01.577982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:01.580071 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
W20260927 12:36:01.580376 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:01.574374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50403 (local address 127.0.98.62:35657)
0927 12:36:01.574525 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:01.574528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:01.574658 (+ 130us) server_negotiation.cc:407] Connection header received
0927 12:36:01.574729 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:01.574731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:01.574781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:01.574861 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:01.575402 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.576003 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.576901 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.577209 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.577298 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:01.580267 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50403: BlockingRecv error: recv got EOF from 127.0.98.2:50403 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 12:36:01 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:01 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:01.779711 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:01.767978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:60723)
0927 12:36:01.768246 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:01.768260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:01.768369 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:01.768725 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:01.768728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:01.768897 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:01.769002 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:01.769008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.769836 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.769839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:01.770384 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:01.770392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.770905 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.770909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:01.770990 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:01.777445 (+ 6455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:01.777461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:01.779565 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
W20260927 12:36:01.779874 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:01.768151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60723 (local address 127.0.98.62:35657)
0927 12:36:01.768296 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:01.768300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:01.768315 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:36:01.768529 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:01.768532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:01.768581 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:01.768661 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:01.769145 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.769720 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.770501 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:01.770777 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:01.770872 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:01.779734 (+ 8862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60723: BlockingRecv error: recv got EOF from 127.0.98.3:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 12:36:02 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:02 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:02.394102 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:02.388311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:58439)
0927 12:36:02.388583 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:02.388596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:02.388681 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:02.388974 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:02.388976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:02.389159 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:02.389263 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:02.389269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.390090 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.390093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:02.390673 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:02.390682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.391195 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.391198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:02.391263 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:02.391737 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:02.391750 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:02.393945 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
W20260927 12:36:02.394222 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:02.388427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58439 (local address 127.0.98.62:35657)
0927 12:36:02.388669 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:02.388672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:02.388686 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:36:02.388777 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:02.388779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:02.388822 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:02.388904 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:02.389415 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.389976 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.390837 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.391078 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.391167 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:02.394097 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58439: BlockingRecv error: recv got EOF from 127.0.98.1:58439 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
Sep 27 12:36:02 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:02 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:02.586879 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:02.581128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:50679)
0927 12:36:02.581460 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:02.581477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:02.581657 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:02.581953 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:02.581956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:02.582160 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:02.582257 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:02.582263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.583018 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.583021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:02.583618 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:02.583627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.584074 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.584078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:02.584210 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:02.584713 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:02.584725 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:02.586741 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
W20260927 12:36:02.587137 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:02.581308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50679 (local address 127.0.98.62:35657)
0927 12:36:02.581466 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:02.581471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:02.581558 (+ 87us) server_negotiation.cc:407] Connection header received
0927 12:36:02.581798 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:02.581800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:02.581847 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:02.581915 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:02.582383 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.582903 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.583740 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.583965 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.584056 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:02.586988 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50679: BlockingRecv error: recv got EOF from 127.0.98.2:50679 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 12:36:02 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:02 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:02.786195 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:02.780633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:37315)
0927 12:36:02.780918 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:02.780933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:02.781028 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:02.781269 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:02.781271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:02.781442 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:02.781536 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:02.781541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.782259 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.782261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:02.782840 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:02.782847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.783303 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.783306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:02.783369 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:02.783877 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:02.783889 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:02.786057 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
W20260927 12:36:02.786286 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:02.780810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37315 (local address 127.0.98.62:35657)
0927 12:36:02.780974 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:02.780979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:02.780998 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:36:02.781117 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:02.781120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:02.781166 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:02.781236 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:02.781646 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.782172 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.782954 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:02.783194 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:02.783276 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:02.786188 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37315: BlockingRecv error: recv got EOF from 127.0.98.3:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 12:36:03 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:03 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:03.401520 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:03.395175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:53649)
0927 12:36:03.395476 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:03.395501 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:03.395620 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:03.395996 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:03.395999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:03.396236 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:03.396335 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:03.396342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.397310 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.397313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:03.398091 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:03.398102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.398640 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.398644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:03.398743 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:03.399296 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:03.399311 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:03.401390 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
W20260927 12:36:03.401669 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:03.395318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53649 (local address 127.0.98.62:35657)
0927 12:36:03.395593 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:03.395597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:03.395612 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:36:03.395743 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:03.395746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:03.395830 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:03.395928 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:03.396453 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.397205 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.398206 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.398514 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.398614 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:03.401544 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53649: BlockingRecv error: recv got EOF from 127.0.98.1:53649 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
Sep 27 12:36:03 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:03 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:03.593222 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:03.587749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:33003)
0927 12:36:03.588150 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:03.588162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:03.588260 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:03.588454 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:03.588455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:03.588616 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:03.588715 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:03.588720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.589471 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.589472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:03.590041 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:03.590048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.590477 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.590479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:03.590533 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:03.591019 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:03.591031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:03.593066 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":72,"threads_started":1}
W20260927 12:36:03.593340 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:03.587815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33003 (local address 127.0.98.62:35657)
0927 12:36:03.587976 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:03.587981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:03.588231 (+ 250us) server_negotiation.cc:407] Connection header received
0927 12:36:03.588323 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:03.588325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:03.588372 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:03.588438 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:03.588848 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.589378 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.590158 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.590383 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.590471 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:03.593226 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33003: BlockingRecv error: recv got EOF from 127.0.98.2:33003 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 12:36:03 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:03 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:03.793088 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:03.787125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:52951)
0927 12:36:03.787380 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:03.787394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:03.787487 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:03.787772 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:03.787774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:03.787975 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:03.788076 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:03.788082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.788925 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.788928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:03.789493 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:03.789501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.790016 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.790019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:03.790110 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:03.790645 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:03.790660 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:03.792931 (+ 2271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
W20260927 12:36:03.793391 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:03.787323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52951 (local address 127.0.98.62:35657)
0927 12:36:03.787449 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:03.787453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:03.787470 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:36:03.787572 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:03.787574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:03.787634 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:03.787729 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:03.788236 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.788784 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.789606 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:03.789891 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:03.789989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:03.793273 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52951: BlockingRecv error: recv got EOF from 127.0.98.3:52951 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 12:36:04 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:04 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:04.409135 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:04.402287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:34121)
0927 12:36:04.402795 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:04.402811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:04.402941 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:04.403271 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:04.403274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:04.403500 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:04.403621 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:04.403629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.404753 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.404756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:04.405364 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:04.405376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.405941 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.405944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:04.406025 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:04.406555 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:04.406568 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:04.409007 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":117,"threads_started":1}
W20260927 12:36:04.409451 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:04.402323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34121 (local address 127.0.98.62:35657)
0927 12:36:04.402628 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:04.402633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:04.402928 (+ 295us) server_negotiation.cc:407] Connection header received
0927 12:36:04.403056 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:04.403060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:04.403133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:04.403208 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:04.403735 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.404610 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.405526 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.405764 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.405890 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:04.409318 (+ 3428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34121: BlockingRecv error: recv got EOF from 127.0.98.1:34121 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
Sep 27 12:36:04 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:04 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:04.599852 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:04.594222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:55595)
0927 12:36:04.594523 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:04.594537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:04.594645 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:04.594870 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:04.594872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:04.595046 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:04.595141 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:04.595146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.595906 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.595908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:04.596511 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:04.596521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.597060 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.597063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:04.597131 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:04.597692 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:04.597704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:04.599633 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
W20260927 12:36:04.599910 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:04.594373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55595 (local address 127.0.98.62:35657)
0927 12:36:04.594526 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:04.594530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:04.594658 (+ 128us) server_negotiation.cc:407] Connection header received
0927 12:36:04.594713 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:04.594715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:04.594764 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:04.594844 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:04.595256 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.595817 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.596645 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.596919 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.597016 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:04.599806 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55595: BlockingRecv error: recv got EOF from 127.0.98.2:55595 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 12:36:04 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:04 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:04.799367 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:04.793915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:51509)
0927 12:36:04.794205 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:04.794220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:04.794325 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:04.794571 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:04.794573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:04.794746 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:04.794841 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:04.794847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.795616 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.795619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:04.796209 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:04.796218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.796654 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.796656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:04.796721 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:04.797210 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:04.797222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:04.799240 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
W20260927 12:36:04.799561 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:04.793996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51509 (local address 127.0.98.62:35657)
0927 12:36:04.794137 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:04.794141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:04.794295 (+ 154us) server_negotiation.cc:407] Connection header received
0927 12:36:04.794423 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:04.794426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:04.794478 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:04.794535 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:04.794963 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.795512 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.796320 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:04.796559 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:04.796644 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:04.799419 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51509: BlockingRecv error: recv got EOF from 127.0.98.3:51509 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:36:05 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:05 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:05.415654 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:05.409994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:49349)
0927 12:36:05.410265 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:05.410279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:05.410382 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:05.410625 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:05.410627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:05.410794 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:05.410888 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:05.410893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.411803 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.411806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:05.412419 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:05.412428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.412881 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.412884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:05.412951 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:05.413433 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:05.413446 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:05.415471 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
W20260927 12:36:05.415781 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:05.410184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49349 (local address 127.0.98.62:35657)
0927 12:36:05.410393 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:05.410396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:05.410444 (+ 48us) server_negotiation.cc:407] Connection header received
0927 12:36:05.410484 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:05.410486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:05.410527 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:05.410596 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:05.411043 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.411655 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.412529 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.412765 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.412849 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:05.415646 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49349: BlockingRecv error: recv got EOF from 127.0.98.1:49349 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
Sep 27 12:36:05 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:05 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:05.606839 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:05.600862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:39655)
0927 12:36:05.601219 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:05.601233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:05.601325 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:05.601673 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:05.601675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:05.601879 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:05.601996 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:05.602003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.603128 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.603132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:05.603702 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:05.603711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.604227 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.604231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:05.604314 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:05.604856 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:05.604869 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:05.606681 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":162,"threads_started":1}
W20260927 12:36:05.606951 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:05.601094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39655 (local address 127.0.98.62:35657)
0927 12:36:05.601254 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:05.601259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:05.601317 (+ 58us) server_negotiation.cc:407] Connection header received
0927 12:36:05.601480 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:05.601484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:05.601539 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:05.601684 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:05.602100 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.603017 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.603817 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.604059 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.604208 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:05.606848 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39655: BlockingRecv error: recv got EOF from 127.0.98.2:39655 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 12:36:05 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:05 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:05.806420 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:05.800246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:47837)
0927 12:36:05.800537 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:05.800551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:05.800736 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:05.801187 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:05.801190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:05.801370 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:05.801472 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:05.801479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.802359 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.802362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:05.802980 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:05.802990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.803448 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.803451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:05.803517 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:05.804052 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:05.804064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:05.806268 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
W20260927 12:36:05.806512 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:05.800364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47837 (local address 127.0.98.62:35657)
0927 12:36:05.800515 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:05.800519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:05.800705 (+ 186us) server_negotiation.cc:407] Connection header received
0927 12:36:05.800868 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:05.800871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:05.801029 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:05.801110 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:05.801666 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.802247 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.803094 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:05.803328 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:05.803416 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:05.806412 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47837: BlockingRecv error: recv got EOF from 127.0.98.3:47837 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 12:36:06 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:06 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:06.422348 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:06.416500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:44247)
0927 12:36:06.416812 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:06.416829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:06.416978 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:06.417224 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:06.417226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:06.417406 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:06.417507 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:06.417513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.418339 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.418343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:06.419043 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:06.419054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.419539 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.419543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:06.419621 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:06.420146 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:06.420158 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:06.422195 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
W20260927 12:36:06.422488 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:06.416627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44247 (local address 127.0.98.62:35657)
0927 12:36:06.416847 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:06.416852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:06.416885 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:36:06.417071 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:06.417073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:06.417119 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:06.417199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:06.417607 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.418224 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.419157 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.419433 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.419523 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:06.422345 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44247: BlockingRecv error: recv got EOF from 127.0.98.1:44247 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":56,"threads_started":1}
Sep 27 12:36:06 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:06 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:06.614629 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:06.608033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:53313)
0927 12:36:06.608397 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:06.608411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:06.608540 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:06.608873 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:06.608876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:06.609056 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:06.609166 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:06.609172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.610076 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.610079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:06.610818 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:06.610831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.611564 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.611568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:06.611669 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:06.612283 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:06.612296 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:06.614470 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
W20260927 12:36:06.614749 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:06.608239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53313 (local address 127.0.98.62:35657)
0927 12:36:06.608396 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:06.608400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:06.608520 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:36:06.608616 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:06.608618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:06.608665 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:06.608742 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:06.609348 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.609901 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.611097 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.611410 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.611517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:06.614648 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53313: BlockingRecv error: recv got EOF from 127.0.98.2:53313 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 12:36:06 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:06 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:06.819604 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:06.807550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:44163)
0927 12:36:06.807913 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:06.807930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:06.808049 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:06.808384 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:06.808387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:06.808558 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:06.808656 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:06.808662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.809437 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.809440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:06.810039 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:06.810048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.810607 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.810610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:06.810677 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:06.817377 (+ 6700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:06.817393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:06.819466 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":156,"threads_started":1}
W20260927 12:36:06.819739 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:06.807666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44163 (local address 127.0.98.62:35657)
0927 12:36:06.807798 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:06.807803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:06.807994 (+ 191us) server_negotiation.cc:407] Connection header received
0927 12:36:06.808179 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:06.808182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:06.808244 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:06.808317 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:06.808782 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.809322 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.810159 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:06.810469 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:06.810554 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:06.819621 (+ 9067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44163: BlockingRecv error: recv got EOF from 127.0.98.3:44163 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 12:36:07 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:07 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:07.430126 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:07.423448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:45679)
0927 12:36:07.423731 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:07.423745 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:07.423843 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:07.424227 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:07.424231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:07.424443 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:07.424575 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:07.424581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.425824 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.425830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:07.426861 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:07.426875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.427549 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.427552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:07.427626 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:07.428229 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:07.428242 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:07.430001 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
W20260927 12:36:07.430361 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:07.423599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45679 (local address 127.0.98.62:35657)
0927 12:36:07.423818 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:07.423822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:07.423839 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:36:07.423923 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:07.423926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:07.423976 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:07.424070 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:07.424736 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.425540 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.427138 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.427380 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.427470 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:07.430220 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45679: BlockingRecv error: recv got EOF from 127.0.98.1:45679 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
Sep 27 12:36:07 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:07 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:07.621510 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:07.615641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:33797)
0927 12:36:07.616014 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:07.616032 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:07.616175 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:07.616503 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:07.616505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:07.616715 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:07.616836 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:07.616844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.617844 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.617848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:07.618502 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:07.618513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.619002 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.619004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:07.619073 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:07.619565 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:07.619576 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:07.621386 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":181,"threads_started":1}
W20260927 12:36:07.621790 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:07.615799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33797 (local address 127.0.98.62:35657)
0927 12:36:07.616019 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:07.616025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:07.616115 (+ 90us) server_negotiation.cc:407] Connection header received
0927 12:36:07.616296 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:07.616299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:07.616352 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:07.616432 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:07.616958 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.617697 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.618641 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.618896 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.618984 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:07.621585 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33797: BlockingRecv error: recv got EOF from 127.0.98.2:33797 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 12:36:07 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:07 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:07.827271 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:07.820945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:50009)
0927 12:36:07.821241 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:07.821255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:07.821380 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:07.821782 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:07.821785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:07.821980 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:07.822081 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:07.822087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.822933 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.822948 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:07.823520 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:07.823529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.824030 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.824034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:07.824168 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:07.824825 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:07.824839 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:07.827139 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
W20260927 12:36:07.827418 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:07.821159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50009 (local address 127.0.98.62:35657)
0927 12:36:07.821335 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:07.821345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:07.821362 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:36:07.821532 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:07.821535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:07.821586 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:07.821669 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:07.822232 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.822803 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.823653 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:07.823892 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:07.824001 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:07.827302 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50009: BlockingRecv error: recv got EOF from 127.0.98.3:50009 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 12:36:08 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:08 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:08.436625 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:08.430979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:35993)
0927 12:36:08.431226 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:08.431240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:08.431334 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:08.431640 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:08.431643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:08.431884 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:08.432005 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:08.432011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.432866 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.432869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:08.433481 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:08.433490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.433953 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.433956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:08.434024 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:08.434522 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:08.434535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:08.436481 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
W20260927 12:36:08.436858 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:08.431104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35993 (local address 127.0.98.62:35657)
0927 12:36:08.431386 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:08.431390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:08.431403 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:36:08.431444 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:08.431446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:08.431488 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:08.431576 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:08.432173 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.432733 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.433601 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.433833 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.433926 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:08.436697 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35993: BlockingRecv error: recv got EOF from 127.0.98.1:35993 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
Sep 27 12:36:08 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:08 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:08.627988 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:08.622420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:44667)
0927 12:36:08.622735 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:08.622751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:08.622887 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:08.623149 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:08.623151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:08.623354 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:08.623450 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:08.623456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.624255 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.624258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:08.624804 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:08.624812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.625255 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.625259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:08.625342 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:08.625872 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:08.625884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:08.627800 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
W20260927 12:36:08.628110 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:08.622571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44667 (local address 127.0.98.62:35657)
0927 12:36:08.622735 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:08.622740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:08.622832 (+ 92us) server_negotiation.cc:407] Connection header received
0927 12:36:08.622983 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:08.622987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:08.623037 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:08.623107 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:08.623574 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.624143 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.624929 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.625149 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.625236 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:08.627973 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44667: BlockingRecv error: recv got EOF from 127.0.98.2:44667 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 12:36:08 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:08 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:08.834147 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:08.828397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:34373)
0927 12:36:08.828657 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:08.828671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:08.828789 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:08.829137 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:08.829140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:08.829377 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:08.829497 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:08.829505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.830428 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.830433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:08.831039 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:08.831049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.831522 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.831526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:08.831597 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:08.832222 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:08.832234 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:08.834024 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
W20260927 12:36:08.834259 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:08.828546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34373 (local address 127.0.98.62:35657)
0927 12:36:08.828710 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:08.828713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:08.828732 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:36:08.828886 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:08.828889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:08.828945 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:08.829061 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:08.829649 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.830284 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.831169 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:08.831396 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:08.831477 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:08.834155 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34373: BlockingRecv error: recv got EOF from 127.0.98.3:34373 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 12:36:09 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:09 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:09.442673 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:09.437453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:43369)
0927 12:36:09.437737 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:09.437751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:09.437846 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:09.438104 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:09.438106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:09.438279 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:09.438373 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:09.438378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.439107 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.439109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:09.439658 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:09.439665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.440199 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.440202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:09.440270 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:09.440809 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:09.440820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:09.442554 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
W20260927 12:36:09.442909 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:09.437597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43369 (local address 127.0.98.62:35657)
0927 12:36:09.437837 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:09.437840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:09.437854 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:36:09.437955 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:09.437958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:09.438002 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:09.438070 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:09.438468 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.439023 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.439771 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.440056 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.440176 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:09.442716 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43369: BlockingRecv error: recv got EOF from 127.0.98.1:43369 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
Sep 27 12:36:09 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:09 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:09.634604 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:09.628949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:46155)
0927 12:36:09.629240 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:09.629255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:09.629359 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:09.629646 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:09.629649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:09.629831 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:09.629931 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:09.629937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.630732 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.630735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:09.631356 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:09.631366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.631902 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.631906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:09.631987 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:09.632555 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:09.632572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:09.634480 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
W20260927 12:36:09.634743 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:09.629071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46155 (local address 127.0.98.62:35657)
0927 12:36:09.629206 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:09.629209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:09.629374 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:36:09.629430 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:09.629433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:09.629497 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:09.629599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:09.630077 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.630600 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.631485 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.631712 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.631797 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:09.634621 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46155: BlockingRecv error: recv got EOF from 127.0.98.2:46155 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 12:36:09 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:09 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:09.840760 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:09.835212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:45667)
0927 12:36:09.835563 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:09.835579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:09.835704 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:09.835967 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:09.835970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:09.836202 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:09.836298 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:09.836303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.837098 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.837102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:09.837663 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:09.837672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.838109 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.838113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:09.838190 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:09.838772 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:09.838785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:09.840608 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
W20260927 12:36:09.840852 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:09.835392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45667 (local address 127.0.98.62:35657)
0927 12:36:09.835562 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:09.835567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:09.835644 (+ 77us) server_negotiation.cc:407] Connection header received
0927 12:36:09.835806 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:09.835809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:09.835858 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:09.835928 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:09.836420 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.836944 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.837779 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:09.838005 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:09.838092 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:09.840752 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45667: BlockingRecv error: recv got EOF from 127.0.98.3:45667 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 12:36:10 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:10 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:10.449302 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:10.443727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:44859)
0927 12:36:10.444040 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:10.444056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:10.444197 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:10.444478 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:10.444482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:10.444681 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:10.444773 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:10.444778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.445561 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.445564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:10.446108 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:10.446115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.446563 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.446567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:10.446649 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:10.447179 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:10.447191 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:10.449132 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
W20260927 12:36:10.449414 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:10.443851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44859 (local address 127.0.98.62:35657)
0927 12:36:10.444137 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:10.444143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:10.444160 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:36:10.444295 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:10.444298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:10.444350 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:10.444435 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:10.444886 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.445452 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.446218 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.446452 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.446538 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:10.449296 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44859: BlockingRecv error: recv got EOF from 127.0.98.1:44859 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
Sep 27 12:36:10 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:10 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:10.641827 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:10.635584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:48659)
0927 12:36:10.635878 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:10.635896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:10.636016 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:10.636349 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:10.636352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:10.636544 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:10.636640 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:10.636646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.637445 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.637448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:10.638001 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:10.638009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.638447 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.638451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:10.638530 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:10.639099 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:10.639112 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:10.641652 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
W20260927 12:36:10.641939 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:10.635793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48659 (local address 127.0.98.62:35657)
0927 12:36:10.635934 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:10.635938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:10.635956 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:36:10.636177 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:10.636179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:10.636225 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:10.636295 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:10.636764 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.637323 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.638119 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.638337 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.638426 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:10.641827 (+ 3401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48659: BlockingRecv error: recv got EOF from 127.0.98.2:48659 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 12:36:10 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:10 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:10.847793 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:10.841794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:53953)
0927 12:36:10.842094 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:10.842108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:10.842212 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:10.842496 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:10.842499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:10.842672 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:10.842770 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:10.842776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.843589 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.843592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:10.844292 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:10.844302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.844899 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.844903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:10.844973 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:10.845528 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:10.845540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:10.847642 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
W20260927 12:36:10.848008 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:10.841908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53953 (local address 127.0.98.62:35657)
0927 12:36:10.842051 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:10.842054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:10.842174 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:36:10.842289 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:10.842291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:10.842340 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:10.842437 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:10.842911 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.843451 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.844458 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:10.844701 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:10.844792 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:10.847809 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53953: BlockingRecv error: recv got EOF from 127.0.98.3:53953 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:36:11 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:11 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:11.456660 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:11.450384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:41011)
0927 12:36:11.450708 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:11.450725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:11.450830 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:11.451150 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:11.451152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:11.451444 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:11.451549 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:11.451555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.452468 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.452471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:11.453039 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:11.453048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.453572 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.453575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:11.453743 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:11.454324 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:11.454338 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:11.456495 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":78,"threads_started":1}
W20260927 12:36:11.456743 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:11.450534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41011 (local address 127.0.98.62:35657)
0927 12:36:11.450880 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:11.450884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:11.450898 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:36:11.450944 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:11.450946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:11.450986 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:11.451069 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:11.451710 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.452319 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.453198 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.453437 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.453536 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:11.456631 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41011: BlockingRecv error: recv got EOF from 127.0.98.1:41011 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":89,"threads_started":1}
Sep 27 12:36:11 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:11 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:11.648846 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:11.642882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:58601)
0927 12:36:11.643185 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:11.643201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:11.643345 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:11.643601 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:11.643604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:11.643814 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:11.643911 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:11.643917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.644801 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.644805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:11.645432 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:11.645443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.645891 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.645895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:11.645976 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:11.646491 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:11.646503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:11.648699 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
W20260927 12:36:11.648967 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:11.642949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58601 (local address 127.0.98.62:35657)
0927 12:36:11.643136 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:11.643142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:11.643306 (+ 164us) server_negotiation.cc:407] Connection header received
0927 12:36:11.643440 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:11.643443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:11.643495 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:11.643579 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:11.644031 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.644605 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.645558 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.645779 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.645869 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:11.648853 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58601: BlockingRecv error: recv got EOF from 127.0.98.2:58601 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 12:36:11 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:11 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:11.859174 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:11.848636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.3:47385)
0927 12:36:11.848971 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:11.848987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:11.849123 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:11.849386 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:11.849388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:11.849583 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:11.849676 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:11.849682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.850423 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.850427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:11.850979 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:11.850987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.851426 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.851430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:11.851504 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:11.857046 (+ 5542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:11.857062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:11.858982 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":87,"threads_started":1}
W20260927 12:36:11.859269 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:11.848808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47385 (local address 127.0.98.62:35657)
0927 12:36:11.848971 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:11.848976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:11.849062 (+ 86us) server_negotiation.cc:407] Connection header received
0927 12:36:11.849214 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:11.849217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:11.849266 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:11.849357 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:11.849790 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.850309 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.851088 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:11.851316 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:11.851410 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:11.859164 (+ 7754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47385: BlockingRecv error: recv got EOF from 127.0.98.3:47385 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 12:36:12 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:12 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:12.463086 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:12.457584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.1:54085)
0927 12:36:12.457839 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:12.457853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:12.457942 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:12.458231 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:12.458233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:12.458419 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:12.458542 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:12.458548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.459395 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.459398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:12.459986 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:12.459995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.460576 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.460579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:12.460651 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:12.461162 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:12.461174 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:12.462965 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
W20260927 12:36:12.463316 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:12.457717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54085 (local address 127.0.98.62:35657)
0927 12:36:12.457973 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:12.457977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:12.457991 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:36:12.458034 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:12.458036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:12.458076 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:12.458156 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:12.458693 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.459240 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.460170 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.460423 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.460525 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:12.463143 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54085: BlockingRecv error: recv got EOF from 127.0.98.1:54085 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":62,"threads_started":1}
Sep 27 12:36:12 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:36:12 dist-test-slave-tzgx krb5kdc[8547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:36:12.655005 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:12.649783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35657 (local address 127.0.98.2:45721)
0927 12:36:12.650041 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:12.650055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:12.650145 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:12.650354 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:12.650357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:12.650536 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:12.650633 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:12.650638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.651318 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.651321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:12.651923 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:12.651932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.652395 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.652398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:12.652469 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:12.653011 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:12.653023 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:12.654875 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35657: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
W20260927 12:36:12.655117 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:12.649912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45721 (local address 127.0.98.62:35657)
0927 12:36:12.650048 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:12.650053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:12.650115 (+ 62us) server_negotiation.cc:407] Connection header received
0927 12:36:12.650216 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:12.650218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:12.650260 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:12.650331 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:12.650723 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.651235 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.652011 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.652269 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.652347 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:12.655014 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45721: BlockingRecv error: recv got EOF from 127.0.98.2:45721 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:36:12.726547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9130
I20260927 12:36:12.731297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9282
I20260927 12:36:12.735836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9414
I20260927 12:36:12.740734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9060
2026-09-27T12:36:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17900 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T12:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:36:12Z Disabled control of system clock
I20260927 12:36:12.771122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33897
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 12:36:12.892526 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:12.892851 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:12.892887 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:12.897642 9610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:36:12.897722 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:12.897737 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:12.897753 9610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:36:12.897768 9610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:36:12.902931 9610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:12.904161 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:12.905628 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:36:12.915119 9610 server_base.cc:1034] running on GCE node
W20260927 12:36:12.915043 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:12.915056 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:12.915043 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:12.916079 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:12.917470 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:12.918644 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790512572918626 us; error 36 us; skew 500 ppm
I20260927 12:36:12.920743 9610 webserver.cc:460] Webserver started at http://127.0.98.62:36857/ using document root <none> and password file <none>
I20260927 12:36:12.921447 9610 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:12.921527 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:12.921904 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:12.924269 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "53b4e85bfb5b4d1b858d0b800a62fafe"
format_stamp: "Formatted at 2026-09-27 12:36:12 on dist-test-slave-tzgx"
server_key: "f827580facae39156ef867c005f47f64"
server_key_iv: "8b842fc3b9352ded186acf94a3bcc8b3"
server_key_version: "encryptionkey@0"
I20260927 12:36:12.924870 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "53b4e85bfb5b4d1b858d0b800a62fafe"
format_stamp: "Formatted at 2026-09-27 12:36:12 on dist-test-slave-tzgx"
server_key: "f827580facae39156ef867c005f47f64"
server_key_iv: "8b842fc3b9352ded186acf94a3bcc8b3"
server_key_version: "encryptionkey@0"
I20260927 12:36:12.928721 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:36:12.931381 9624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:12.932844 9610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:36:12.933018 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "53b4e85bfb5b4d1b858d0b800a62fafe"
format_stamp: "Formatted at 2026-09-27 12:36:12 on dist-test-slave-tzgx"
server_key: "f827580facae39156ef867c005f47f64"
server_key_iv: "8b842fc3b9352ded186acf94a3bcc8b3"
server_key_version: "encryptionkey@0"
I20260927 12:36:12.933146 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:12.956573 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:12.957355 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:12.957552 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:12.964681 9610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33897
I20260927 12:36:12.964859 9676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33897 every 8 connection(s)
I20260927 12:36:12.965757 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:36:12.968120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9610
I20260927 12:36:12.968232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:36:12.968540 392 external_mini_cluster.cc:1428] Setting key d20d72258684133f44d24dea2fde554e
I20260927 12:36:12.969662 9677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:36:12.976444 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe: Bootstrap starting.
I20260927 12:36:12.978506 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:12.970039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.62:33897)
0927 12:36:12.970733 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:12.970749 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:12.970786 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:36:12.971463 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:12.971481 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:12.971915 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:12.972290 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:12.972746 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.973816 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.974868 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:12.975436 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:12.975611 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:12.976648 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:12.976655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:12.976676 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:12.977117 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:12.977175 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:12.977197 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:12.977288 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:12.977461 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 12:36:12.977740 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260927 12:36:12.978982 9677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe: Neither blocks nor log segments found. Creating new log.
I20260927 12:36:12.979933 9677 log.cc:824] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe: Log is configured to *not* fsync() on all Append() calls
I20260927 12:36:12.982136 9677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe: No bootstrap required, opened a new log
I20260927 12:36:12.985937 9677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33897 } }
I20260927 12:36:12.986197 9677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:36:12.986232 9677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b4e85bfb5b4d1b858d0b800a62fafe, State: Initialized, Role: FOLLOWER
I20260927 12:36:12.986676 9677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33897 } }
I20260927 12:36:12.986812 9677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:36:12.986857 9677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:36:12.986958 9677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:36:12.987915 9677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33897 } }
I20260927 12:36:12.988297 9677 leader_election.cc:302] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53b4e85bfb5b4d1b858d0b800a62fafe; no voters:
I20260927 12:36:12.988608 9677 leader_election.cc:288] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:36:12.988757 9682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:36:12.988976 9682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [term 1 LEADER]: Becoming Leader. State: Replica: 53b4e85bfb5b4d1b858d0b800a62fafe, State: Running, Role: LEADER
I20260927 12:36:12.989357 9682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33897 } }
I20260927 12:36:12.989965 9677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:36:12.991014 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33897 } } }
I20260927 12:36:12.991144 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [sys.catalog]: This master's current role is: LEADER
I20260927 12:36:12.991431 9684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53b4e85bfb5b4d1b858d0b800a62fafe. Latest consensus state: current_term: 1 leader_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b4e85bfb5b4d1b858d0b800a62fafe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33897 } } }
I20260927 12:36:12.991525 9684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe [sys.catalog]: This master's current role is: LEADER
I20260927 12:36:12.991529 9691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:36:12.994374 9691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:36:13.000851 9691 catalog_manager.cc:1347] Generated new cluster ID: 3130c033253747c591cf01ac678de94c
I20260927 12:36:13.000952 9691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:36:13.031924 9691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:36:13.033282 9691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:36:13.042637 9691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53b4e85bfb5b4d1b858d0b800a62fafe: Generated new TSK 0
I20260927 12:36:13.043543 9691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:36:13.057315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33897
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 12:36:13.181311 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:13.181618 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:13.181668 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:13.185981 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:13.186057 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:13.186133 9701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:36:13.191529 9701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:13.192726 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:13.194217 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:13.204280 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.204254 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.204288 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:13.204623 9701 server_base.cc:1034] running on GCE node
I20260927 12:36:13.205111 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:13.205804 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:13.207000 9701 hybrid_clock.cc:648] HybridClock initialized: now 1790512573206968 us; error 47 us; skew 500 ppm
I20260927 12:36:13.209260 9701 webserver.cc:460] Webserver started at http://127.0.98.1:38631/ using document root <none> and password file <none>
I20260927 12:36:13.209892 9701 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:13.209975 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:13.210180 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:13.213050 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "3dc45852a30e4dcda49abfa51538eaf8"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "d3e90712f93ad6d89ba4386cc887c4c7"
server_key_iv: "eac4d6fb2ee7a2010f2f1392c6dc1db0"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.213845 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc45852a30e4dcda49abfa51538eaf8"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "d3e90712f93ad6d89ba4386cc887c4c7"
server_key_iv: "eac4d6fb2ee7a2010f2f1392c6dc1db0"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.218389 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:36:13.221355 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.222977 9701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:36:13.223119 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "3dc45852a30e4dcda49abfa51538eaf8"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "d3e90712f93ad6d89ba4386cc887c4c7"
server_key_iv: "eac4d6fb2ee7a2010f2f1392c6dc1db0"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.223245 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:13.242033 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:13.242839 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:13.243022 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:13.243618 9701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:13.244760 9701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:13.244819 9701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.244879 9701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:13.244939 9701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.254853 9701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36983
I20260927 12:36:13.254905 9828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36983 every 8 connection(s)
I20260927 12:36:13.256340 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:36:13.265058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9701
I20260927 12:36:13.265172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:36:13.265446 392 external_mini_cluster.cc:1428] Setting key f9c32d38d310fcf2b18e1246e2adeeed
I20260927 12:36:13.267204 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.258915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50515 (local address 127.0.98.62:33897)
0927 12:36:13.259053 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:13.259057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:13.260233 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 12:36:13.261695 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:13.261698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:13.261757 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:13.261841 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:13.263003 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.263576 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.265075 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.265355 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.265457 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:13.266725 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:13.266729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:13.266732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:13.266745 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:13.266771 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:13.266781 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:13.266863 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:13.267019 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 12:36:13.267089 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:36:13.267642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33897
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 12:36:13.268498 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.259281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33897 (local address 127.0.98.1:50515)
0927 12:36:13.260026 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:13.260070 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:13.261442 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:13.262067 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:13.262075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:13.262173 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:13.262798 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:13.262815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.263774 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.263778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:13.264896 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:13.264911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.265551 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.265555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:13.265786 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:13.266430 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:36:13.266432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:13.266493 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:36:13.266571 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:13.266867 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:13.266878 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:13.266885 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:36:13.267007 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 12:36:13.267339 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260927 12:36:13.270112 9829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33897
I20260927 12:36:13.270416 9829 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:13.270996 9829 heartbeater.cc:511] Master 127.0.98.62:33897 requested a full tablet report, sending...
I20260927 12:36:13.273016 9641 ts_manager.cc:194] Registered new tserver with Master: 3dc45852a30e4dcda49abfa51538eaf8 (127.0.98.1:36983)
I20260927 12:36:13.274766 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50515
W20260927 12:36:13.392728 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:13.393054 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:13.393093 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:13.397818 9832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:13.397912 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:13.397980 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:36:13.403250 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:13.404403 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:13.405819 9832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:36:13.416105 9832 server_base.cc:1034] running on GCE node
W20260927 12:36:13.416020 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.416118 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.416020 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:13.416747 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:13.417392 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:13.418562 9832 hybrid_clock.cc:648] HybridClock initialized: now 1790512573418541 us; error 33 us; skew 500 ppm
I20260927 12:36:13.420511 9832 webserver.cc:460] Webserver started at http://127.0.98.2:42721/ using document root <none> and password file <none>
I20260927 12:36:13.421082 9832 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:13.421146 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:13.421334 9832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:13.423457 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "87aff06287f740d28d858d0c72843815"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "d1a9c2f4e48ce06a5083aabfe5df6a7f"
server_key_iv: "d41b2dcf1b9afb87dea6968771a6f761"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.424243 9832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "87aff06287f740d28d858d0c72843815"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "d1a9c2f4e48ce06a5083aabfe5df6a7f"
server_key_iv: "d41b2dcf1b9afb87dea6968771a6f761"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.428524 9832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:36:13.431557 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.433060 9832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:36:13.433197 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "87aff06287f740d28d858d0c72843815"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "d1a9c2f4e48ce06a5083aabfe5df6a7f"
server_key_iv: "d41b2dcf1b9afb87dea6968771a6f761"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.433350 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:13.457170 9832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:13.457943 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:13.458146 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:13.458788 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:13.459857 9832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:13.459924 9832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.459982 9832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:13.460016 9832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.468927 9832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39399
I20260927 12:36:13.468990 9959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39399 every 8 connection(s)
I20260927 12:36:13.469883 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:36:13.475477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9832
I20260927 12:36:13.475582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:36:13.475884 392 external_mini_cluster.cc:1428] Setting key fb83e8decea6ca407aa98095cff54055
I20260927 12:36:13.478142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33897
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 12:36:13.480391 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.471862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54773 (local address 127.0.98.62:33897)
0927 12:36:13.472017 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:13.472021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:13.473304 (+ 1283us) server_negotiation.cc:407] Connection header received
0927 12:36:13.474748 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:13.474752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:13.474807 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:13.474889 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:13.475980 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.476646 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.478089 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.478414 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.478539 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:13.479894 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:13.479897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:13.479899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:13.479912 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:13.479934 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:13.479942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:13.480008 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:13.480197 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 12:36:13.480276 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:36:13.481459 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.472324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33897 (local address 127.0.98.2:54773)
0927 12:36:13.473144 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:13.473192 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:13.474500 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:13.475062 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:13.475070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:13.475165 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:13.475816 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:13.475834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.476817 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.476821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:13.477964 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:13.477976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.478600 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.478604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:13.478772 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:13.479546 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:36:13.479548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:13.479611 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:36:13.479709 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:13.480025 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:13.480033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:13.480039 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:36:13.480189 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 12:36:13.480541 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":143,"threads_started":1}
I20260927 12:36:13.482733 9960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33897
I20260927 12:36:13.483047 9960 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:13.483809 9960 heartbeater.cc:511] Master 127.0.98.62:33897 requested a full tablet report, sending...
I20260927 12:36:13.485344 9641 ts_manager.cc:194] Registered new tserver with Master: 87aff06287f740d28d858d0c72843815 (127.0.98.2:39399)
I20260927 12:36:13.485944 9641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54773
W20260927 12:36:13.600521 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:13.601006 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:13.601078 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:13.607288 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:13.607389 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:13.607456 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:36:13.614377 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:13.616011 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:13.618013 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:13.629271 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.629280 9971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.629540 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:13.629838 9963 server_base.cc:1034] running on GCE node
I20260927 12:36:13.630350 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:13.631121 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:13.632326 9963 hybrid_clock.cc:648] HybridClock initialized: now 1790512573632295 us; error 52 us; skew 500 ppm
I20260927 12:36:13.634665 9963 webserver.cc:460] Webserver started at http://127.0.98.3:35719/ using document root <none> and password file <none>
I20260927 12:36:13.635610 9963 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:13.635695 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:13.635910 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:13.638447 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "331f2d39d2ea4edabab6d10325c45f41"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "484a82ac58d6e23b1603cf2a67aaa97c"
server_key_iv: "14f026767a86698faeab90091d663097"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.639158 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "331f2d39d2ea4edabab6d10325c45f41"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "484a82ac58d6e23b1603cf2a67aaa97c"
server_key_iv: "14f026767a86698faeab90091d663097"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.643486 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:36:13.646617 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.648252 9963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:36:13.648406 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "331f2d39d2ea4edabab6d10325c45f41"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "484a82ac58d6e23b1603cf2a67aaa97c"
server_key_iv: "14f026767a86698faeab90091d663097"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.648550 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:13.662909 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:13.663647 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:13.663829 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:13.664522 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:13.665550 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:13.665598 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.665638 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:13.665659 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.675091 9963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33725
I20260927 12:36:13.675120 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33725 every 8 connection(s)
I20260927 12:36:13.676069 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:36:13.685598 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.677976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56745 (local address 127.0.98.62:33897)
0927 12:36:13.678101 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:13.678105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:13.679344 (+ 1239us) server_negotiation.cc:407] Connection header received
0927 12:36:13.680867 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:13.680870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:13.680941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:13.681017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:13.682000 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.682525 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.683693 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.683913 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.684015 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:13.685129 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:13.685133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:13.685136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:13.685151 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:13.685171 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:13.685178 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:13.685238 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:13.685375 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 12:36:13.685441 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:36:13.685901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 9963
I20260927 12:36:13.686079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:36:13.686372 392 external_mini_cluster.cc:1428] Setting key 6260a88672fcc8113c29e5004d808356
I20260927 12:36:13.686678 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.678399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33897 (local address 127.0.98.3:56745)
0927 12:36:13.679169 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:13.679215 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:13.680638 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:13.681210 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:13.681227 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:13.681325 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:13.681854 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:13.681868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.682648 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.682651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:13.683572 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:13.683582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:13.684072 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:13.684076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:13.684268 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:13.684858 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:36:13.684859 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:13.684902 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:36:13.684977 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:13.685245 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:13.685253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:13.685259 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:36:13.685369 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:36:13.685717 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":129,"threads_started":1}
I20260927 12:36:13.687901 10091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33897
I20260927 12:36:13.688299 10091 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:13.688941 10091 heartbeater.cc:511] Master 127.0.98.62:33897 requested a full tablet report, sending...
I20260927 12:36:13.690080 9640 ts_manager.cc:194] Registered new tserver with Master: 331f2d39d2ea4edabab6d10325c45f41 (127.0.98.3:33725)
I20260927 12:36:13.690732 9640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56745
I20260927 12:36:13.701373 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:36:13.704558 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:13.703484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.62:33897)
0927 12:36:13.703824 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:13.703836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:13.703853 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:36:13.703896 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:13.703898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:13.703950 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:13.704040 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:13.704404 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53990: BlockingRecv error: recv got EOF from 127.0.0.1:53990 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:36:13.705056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9701
I20260927 12:36:13.710358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9832
I20260927 12:36:13.715490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9963
I20260927 12:36:13.720428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 9610
2026-09-27T12:36:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (978 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T12:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:36:13Z Disabled control of system clock
I20260927 12:36:13.748737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45509
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:36:13.874392 10102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:13.874723 10102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:13.874764 10102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:13.879159 10102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:36:13.879253 10102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:13.879267 10102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:13.879283 10102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:36:13.879298 10102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:36:13.884670 10102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:13.885823 10102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:13.887241 10102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:13.896404 10108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:13.896340 10110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:13.896421 10102 server_base.cc:1034] running on GCE node
W20260927 12:36:13.896346 10107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:13.897282 10102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:13.898494 10102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:13.899691 10102 hybrid_clock.cc:648] HybridClock initialized: now 1790512573899645 us; error 62 us; skew 500 ppm
I20260927 12:36:13.901836 10102 webserver.cc:460] Webserver started at http://127.0.98.62:36147/ using document root <none> and password file <none>
I20260927 12:36:13.902486 10102 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:13.902561 10102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:13.902740 10102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:13.905207 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "38865b322b984abfb2846a970c1dc1d0"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "21f480649240b7919d8659cbce00a949"
server_key_iv: "4e16dfb6d6fcb896b8447693e01dac5b"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.905819 10102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "38865b322b984abfb2846a970c1dc1d0"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "21f480649240b7919d8659cbce00a949"
server_key_iv: "4e16dfb6d6fcb896b8447693e01dac5b"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.909902 10102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:36:13.912734 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:13.914304 10102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:36:13.914444 10102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "38865b322b984abfb2846a970c1dc1d0"
format_stamp: "Formatted at 2026-09-27 12:36:13 on dist-test-slave-tzgx"
server_key: "21f480649240b7919d8659cbce00a949"
server_key_iv: "4e16dfb6d6fcb896b8447693e01dac5b"
server_key_version: "encryptionkey@0"
I20260927 12:36:13.914600 10102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:13.919919 10102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:13.920868 10102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:13.921070 10102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:13.931929 10102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45509
I20260927 12:36:13.931951 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45509 every 8 connection(s)
I20260927 12:36:13.933274 10102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:36:13.935735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10102
I20260927 12:36:13.935859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:36:13.936182 392 external_mini_cluster.cc:1428] Setting key 0bdeaa4eb86a9dbbb7ac73e1e42a8363
I20260927 12:36:13.937536 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:36:13.942699 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:13.938067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.98.62:45509)
0927 12:36:13.938818 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:13.938833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:13.938876 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:36:13.939744 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:13.939765 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:13.940198 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:13.940544 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:13.941374 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:13.941379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:13.941392 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:13.941403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:13.941461 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:13.941478 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:13.941611 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:13.941678 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 12:36:13.941928 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":174,"threads_started":1}
I20260927 12:36:13.944684 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0: Bootstrap starting.
I20260927 12:36:13.947196 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0: Neither blocks nor log segments found. Creating new log.
I20260927 12:36:13.948056 10169 log.cc:824] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:36:13.951020 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0: No bootstrap required, opened a new log
I20260927 12:36:13.955487 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38865b322b984abfb2846a970c1dc1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260927 12:36:13.955783 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:36:13.955837 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38865b322b984abfb2846a970c1dc1d0, State: Initialized, Role: FOLLOWER
I20260927 12:36:13.956434 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38865b322b984abfb2846a970c1dc1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260927 12:36:13.956684 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:36:13.956784 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:36:13.956916 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:36:13.958087 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38865b322b984abfb2846a970c1dc1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260927 12:36:13.958493 10169 leader_election.cc:302] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38865b322b984abfb2846a970c1dc1d0; no voters:
I20260927 12:36:13.958793 10169 leader_election.cc:288] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:36:13.958978 10174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:36:13.959187 10174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 38865b322b984abfb2846a970c1dc1d0, State: Running, Role: LEADER
I20260927 12:36:13.959559 10174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38865b322b984abfb2846a970c1dc1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } }
I20260927 12:36:13.960201 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:36:13.961079 10175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38865b322b984abfb2846a970c1dc1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38865b322b984abfb2846a970c1dc1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } }
I20260927 12:36:13.961325 10175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:36:13.961824 10176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38865b322b984abfb2846a970c1dc1d0. Latest consensus state: current_term: 1 leader_uuid: "38865b322b984abfb2846a970c1dc1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38865b322b984abfb2846a970c1dc1d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45509 } } }
I20260927 12:36:13.961990 10176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:36:13.962293 10183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:36:13.967911 10183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:36:13.975843 10183 catalog_manager.cc:1347] Generated new cluster ID: ed8fc2f979484f3e85d3461b39569284
I20260927 12:36:13.975952 10183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:36:13.991456 10183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:36:13.992254 10183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:36:14.010043 10183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38865b322b984abfb2846a970c1dc1d0: Generated new TSK 0
I20260927 12:36:14.011094 10183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:36:14.024861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--builtin_ntp_servers=127.0.98.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:36:14.192963 10193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:14.193270 10193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:14.193320 10193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:14.197749 10193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:14.197834 10193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:14.197894 10193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:36:14.203088 10193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:14.204322 10193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:14.206159 10193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:36:14.218278 10193 server_base.cc:1034] running on GCE node
W20260927 12:36:14.218181 10199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:14.218194 10201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:14.218181 10198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:14.218911 10193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:14.219651 10193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:14.220872 10193 hybrid_clock.cc:648] HybridClock initialized: now 1790512574220836 us; error 49 us; skew 500 ppm
I20260927 12:36:14.222863 10193 webserver.cc:460] Webserver started at http://127.0.98.1:46619/ using document root <none> and password file <none>
I20260927 12:36:14.223454 10193 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:14.223528 10193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:14.223726 10193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:14.226227 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb8b2985939044e8b310862f60e7f667"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "6c6b5be020a355091b77b4ec5d4d6a28"
server_key_iv: "bc59bab8338bb00a27b335a97a79c6fc"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.226902 10193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb8b2985939044e8b310862f60e7f667"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "6c6b5be020a355091b77b4ec5d4d6a28"
server_key_iv: "bc59bab8338bb00a27b335a97a79c6fc"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.231211 10193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:36:14.234182 10207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.235589 10193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:36:14.235723 10193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "fb8b2985939044e8b310862f60e7f667"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "6c6b5be020a355091b77b4ec5d4d6a28"
server_key_iv: "bc59bab8338bb00a27b335a97a79c6fc"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.235852 10193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:14.240777 10193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:14.241503 10193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:14.241658 10193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:14.242201 10193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:14.243134 10193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:14.243180 10193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.243219 10193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:14.243238 10193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.251966 10193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42969
I20260927 12:36:14.252055 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42969 every 8 connection(s)
I20260927 12:36:14.253014 10193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:36:14.254544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10193
I20260927 12:36:14.254655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:36:14.254930 392 external_mini_cluster.cc:1428] Setting key 464171ca0a897f23315d9ec677674002
I20260927 12:36:14.257079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--builtin_ntp_servers=127.0.98.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 12:36:14.260190 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:14.255234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44713 (local address 127.0.98.62:45509)
0927 12:36:14.255354 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:14.255357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:14.256841 (+ 1484us) server_negotiation.cc:407] Connection header received
0927 12:36:14.258443 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:14.258446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:14.258491 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:14.258596 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:14.259655 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:14.259659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:14.259662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:14.259675 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:14.259698 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:14.259707 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:14.259798 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:14.259981 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 12:36:14.260024 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 12:36:14.261350 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:14.255777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.1:44713)
0927 12:36:14.256656 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:14.256715 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:14.258204 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:14.258745 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:14.258752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:14.258843 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:14.259390 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:36:14.259391 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:14.259435 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:36:14.259504 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:14.259835 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:14.259842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:14.259847 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:36:14.260008 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 12:36:14.260402 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":158,"threads_started":1}
I20260927 12:36:14.262805 10321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260927 12:36:14.263106 10321 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:14.263659 10321 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260927 12:36:14.265483 10127 ts_manager.cc:194] Registered new tserver with Master: fb8b2985939044e8b310862f60e7f667 (127.0.98.1:42969)
W20260927 12:36:14.379922 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:14.380294 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:14.380350 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:14.384709 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:14.384786 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:14.384846 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:36:14.390206 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:14.391378 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:14.392906 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:14.403462 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:14.403599 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:14.403654 10324 server_base.cc:1034] running on GCE node
W20260927 12:36:14.403451 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:14.404765 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:14.405661 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:14.406883 10324 hybrid_clock.cc:648] HybridClock initialized: now 1790512574406840 us; error 59 us; skew 500 ppm
I20260927 12:36:14.410394 10324 webserver.cc:460] Webserver started at http://127.0.98.2:42155/ using document root <none> and password file <none>
I20260927 12:36:14.411442 10324 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:14.411552 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:14.411753 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:14.414405 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "a6b86564df8340b8b2348009c836be2b"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "2c7e958312103261fe70819b724ca1b5"
server_key_iv: "a9593a43fe599d2fa46a5ef89932c28f"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.415077 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a6b86564df8340b8b2348009c836be2b"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "2c7e958312103261fe70819b724ca1b5"
server_key_iv: "a9593a43fe599d2fa46a5ef89932c28f"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.419382 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 12:36:14.422160 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.423581 10324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:36:14.423710 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "a6b86564df8340b8b2348009c836be2b"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "2c7e958312103261fe70819b724ca1b5"
server_key_iv: "a9593a43fe599d2fa46a5ef89932c28f"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.423861 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:14.428742 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:14.429401 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:14.429581 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:14.430112 10324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:14.431037 10324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:14.431085 10324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.431124 10324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:14.431149 10324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.441226 10324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40251
I20260927 12:36:14.441274 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40251 every 8 connection(s)
I20260927 12:36:14.442320 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:36:14.444346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10324
I20260927 12:36:14.444473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:36:14.444761 392 external_mini_cluster.cc:1428] Setting key 0654bfa9383a184bd45aabb158668b9f
I20260927 12:36:14.446852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--builtin_ntp_servers=127.0.98.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 12:36:14.449335 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:14.444345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51439 (local address 127.0.98.62:45509)
0927 12:36:14.444481 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:14.444485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:14.446020 (+ 1535us) server_negotiation.cc:407] Connection header received
0927 12:36:14.447470 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:14.447474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:14.447518 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:14.447588 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:14.448814 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:14.448817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:14.448819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:14.448831 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:14.448854 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:14.448862 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:14.448975 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:14.449155 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 12:36:14.449198 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:36:14.450464 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:14.444909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.2:51439)
0927 12:36:14.445839 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:14.445900 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:14.447242 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:14.447777 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:14.447784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:14.447902 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:14.448501 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:36:14.448503 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:14.448559 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:36:14.448654 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:14.448980 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:14.448987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:14.448993 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:36:14.449173 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 12:36:14.449440 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":161,"threads_started":1}
I20260927 12:36:14.451701 10452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260927 12:36:14.451972 10452 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:14.452536 10452 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260927 12:36:14.453853 10127 ts_manager.cc:194] Registered new tserver with Master: a6b86564df8340b8b2348009c836be2b (127.0.98.2:40251)
W20260927 12:36:14.566293 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:14.566632 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:14.566699 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:14.571172 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:14.571270 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:14.571357 10455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:36:14.576676 10455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:14.577965 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:14.579710 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:14.590462 10463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:14.590466 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:14.590466 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:14.591205 10455 server_base.cc:1034] running on GCE node
I20260927 12:36:14.591687 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:14.592501 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:14.593705 10455 hybrid_clock.cc:648] HybridClock initialized: now 1790512574593646 us; error 75 us; skew 500 ppm
I20260927 12:36:14.596447 10455 webserver.cc:460] Webserver started at http://127.0.98.3:37591/ using document root <none> and password file <none>
I20260927 12:36:14.597085 10455 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:14.597159 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:14.597383 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:14.599998 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a76bb4642f746e29f28023c149d41b1"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "b711aa7eb3493949c364804cc9cf7ec8"
server_key_iv: "12d1d1e946f272646df1a0d5bf13a52f"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.600716 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a76bb4642f746e29f28023c149d41b1"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "b711aa7eb3493949c364804cc9cf7ec8"
server_key_iv: "12d1d1e946f272646df1a0d5bf13a52f"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.605132 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:36:14.608319 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.609961 10455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:36:14.610126 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "6a76bb4642f746e29f28023c149d41b1"
format_stamp: "Formatted at 2026-09-27 12:36:14 on dist-test-slave-tzgx"
server_key: "b711aa7eb3493949c364804cc9cf7ec8"
server_key_iv: "12d1d1e946f272646df1a0d5bf13a52f"
server_key_version: "encryptionkey@0"
I20260927 12:36:14.610277 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:14.615633 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:14.616369 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:14.616542 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:14.617125 10455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:14.618136 10455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:14.618184 10455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.618238 10455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:14.618263 10455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:14.627753 10455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42061
I20260927 12:36:14.627781 10582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42061 every 8 connection(s)
I20260927 12:36:14.628747 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:36:14.634387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10455
I20260927 12:36:14.634509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:36:14.634788 392 external_mini_cluster.cc:1428] Setting key 9d3b805499631363e94eaa66e3e554e2
I20260927 12:36:14.635480 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:14.630665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57101 (local address 127.0.98.62:45509)
0927 12:36:14.630817 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:14.630821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:14.631894 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 12:36:14.633318 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:14.633322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:14.633365 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:14.633448 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:14.634928 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:14.634932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:14.634936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:36:14.634950 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:14.634974 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:36:14.634984 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:14.635083 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:14.635283 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:36:14.635335 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:36:14.636595 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:14.631034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45509 (local address 127.0.98.3:57101)
0927 12:36:14.631731 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:14.631776 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:14.633049 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:14.633710 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:14.633718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:14.633858 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:14.634517 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:36:14.634519 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:14.634591 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:36:14.634688 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:14.635081 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:14.635092 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:14.635099 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:36:14.635310 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 12:36:14.635582 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260927 12:36:14.637686 10583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45509
I20260927 12:36:14.637909 10583 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:14.638341 10583 heartbeater.cc:511] Master 127.0.98.62:45509 requested a full tablet report, sending...
I20260927 12:36:14.639582 10127 ts_manager.cc:194] Registered new tserver with Master: 6a76bb4642f746e29f28023c149d41b1 (127.0.98.3:42061)
I20260927 12:36:14.649646 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:36:14.652563 10172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:36:14.651659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.0.98.62:45509)
0927 12:36:14.651844 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:14.651848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:14.651929 (+ 81us) server_negotiation.cc:407] Connection header received
0927 12:36:14.652068 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:14.652070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:14.652127 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:14.652201 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:14.652406 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46502: BlockingRecv error: recv got EOF from 127.0.0.1:46502 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:36:14.652670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10193
I20260927 12:36:14.657671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10324
I20260927 12:36:14.662304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10455
I20260927 12:36:14.666908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10102
2026-09-27T12:36:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (946 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:36:14 dist-test-slave-tzgx krb5kdc[10593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:36:14 dist-test-slave-tzgx krb5kdc[10593](info): set up 2 sockets
Sep 27 12:36:14 dist-test-slave-tzgx krb5kdc[10593](info): commencing operation
krb5kdc: starting...
W20260927 12:36:15.640763 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:36:15 dist-test-slave-tzgx krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:36:15.768720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:36:15.889796 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:15.890086 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:15.890128 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:15.894310 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:36:15.894385 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:15.894404 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:15.894420 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:36:15.894434 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:36:15.900346 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:15.901489 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:15.902526 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:15.912215 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:15.912405 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:15.912528 10606 server_base.cc:1034] running on GCE node
W20260927 12:36:15.912407 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:15.913360 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:15.914628 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:15.915815 10606 hybrid_clock.cc:648] HybridClock initialized: now 1790512575915793 us; error 42 us; skew 500 ppm
Sep 27 12:36:15 dist-test-slave-tzgx krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:36:15.919181 10606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:36:15.920594 10606 webserver.cc:460] Webserver started at http://127.0.98.62:35363/ using document root <none> and password file <none>
I20260927 12:36:15.921335 10606 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:15.921433 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:15.921689 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:15.924615 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "320256de75bc4732a8655369e99228a4"
format_stamp: "Formatted at 2026-09-27 12:36:15 on dist-test-slave-tzgx"
server_key: "4b906e2cf92e97bce9ecafdec8dd0e87"
server_key_iv: "ffb4edb32e917b32c8453022c83dec6f"
server_key_version: "encryptionkey@0"
I20260927 12:36:15.925292 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "320256de75bc4732a8655369e99228a4"
format_stamp: "Formatted at 2026-09-27 12:36:15 on dist-test-slave-tzgx"
server_key: "4b906e2cf92e97bce9ecafdec8dd0e87"
server_key_iv: "ffb4edb32e917b32c8453022c83dec6f"
server_key_version: "encryptionkey@0"
I20260927 12:36:15.929563 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:36:15.932369 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:15.933674 10606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:36:15.933821 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "320256de75bc4732a8655369e99228a4"
format_stamp: "Formatted at 2026-09-27 12:36:15 on dist-test-slave-tzgx"
server_key: "4b906e2cf92e97bce9ecafdec8dd0e87"
server_key_iv: "ffb4edb32e917b32c8453022c83dec6f"
server_key_version: "encryptionkey@0"
I20260927 12:36:15.933933 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:15.948405 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:15.951623 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:15.951826 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:15.958706 10606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37243
I20260927 12:36:15.958981 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37243 every 8 connection(s)
I20260927 12:36:15.959735 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:36:15.962469 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:36:15.965829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10606
I20260927 12:36:15.965972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:36:15.966266 392 external_mini_cluster.cc:1428] Setting key 61ba4406d304bd96c3c685f4e2f724ad
I20260927 12:36:15.968814 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4: Bootstrap starting.
I20260927 12:36:15.971086 10674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4: Neither blocks nor log segments found. Creating new log.
I20260927 12:36:15.971798 10674 log.cc:824] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:36:15.973830 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4: No bootstrap required, opened a new log
Sep 27 12:36:15 dist-test-slave-tzgx krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:36:15.977418 10674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320256de75bc4732a8655369e99228a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37243 } }
I20260927 12:36:15.977679 10674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:36:15.977749 10674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320256de75bc4732a8655369e99228a4, State: Initialized, Role: FOLLOWER
I20260927 12:36:15.978214 10674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320256de75bc4732a8655369e99228a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37243 } }
I20260927 12:36:15.978365 10674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:36:15.978436 10674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:36:15.978539 10674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:36:15.979565 10674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320256de75bc4732a8655369e99228a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37243 } }
I20260927 12:36:15.979923 10674 leader_election.cc:302] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320256de75bc4732a8655369e99228a4; no voters:
I20260927 12:36:15.980329 10674 leader_election.cc:288] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:36:15.980532 10679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:36:15.980727 10679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [term 1 LEADER]: Becoming Leader. State: Replica: 320256de75bc4732a8655369e99228a4, State: Running, Role: LEADER
I20260927 12:36:15.981122 10679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320256de75bc4732a8655369e99228a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37243 } }
I20260927 12:36:15.981642 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:15.967825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.98.62:37243)
0927 12:36:15.968314 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:15.968329 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:15.968365 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:36:15.969068 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:15.969087 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:15.969435 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:15.970482 (+ 1047us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:15.970806 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:15.971851 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:15.972775 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:15.973233 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:15.973383 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:15.976038 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:15.976044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:15.976057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:36:15.976084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:15.978209 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:15.978742 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:15.978749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:15.978756 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:15.978891 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:15.979194 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:15.979197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:15.979198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:15.979624 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:15.979810 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:15.980080 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:36:15.980399 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":146,"threads_started":1}
I20260927 12:36:15.981742 10674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:36:15.982659 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320256de75bc4732a8655369e99228a4. Latest consensus state: current_term: 1 leader_uuid: "320256de75bc4732a8655369e99228a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320256de75bc4732a8655369e99228a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37243 } } }
I20260927 12:36:15.982785 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:36:15.983069 10680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320256de75bc4732a8655369e99228a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320256de75bc4732a8655369e99228a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37243 } } }
I20260927 12:36:15.983161 10680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:36:15.983412 10686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:36:15.986588 10686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:36:15.992587 10686 catalog_manager.cc:1347] Generated new cluster ID: 1feeff9573d2402ab026db9269579802
I20260927 12:36:15.992674 10686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:36:16.013381 10686 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:36:16.014391 10686 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:36:16.020509 10686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4: Generated new TSK 0
I20260927 12:36:16.021304 10686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:36:16.063122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37243
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:36:16.184413 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:16.184762 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:16.184839 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:16.189280 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:16.189368 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:16.189455 10700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:36:16.195824 10700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:16.197073 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:16.198202 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:16.209570 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:16.209615 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:16.209649 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:16.209867 10700 server_base.cc:1034] running on GCE node
I20260927 12:36:16.210500 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:16.211238 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:16.212463 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790512576212444 us; error 31 us; skew 500 ppm
Sep 27 12:36:16 dist-test-slave-tzgx krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:36:16.215822 10700 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:36:16.216842 10700 webserver.cc:460] Webserver started at http://127.0.98.1:42083/ using document root <none> and password file <none>
I20260927 12:36:16.217407 10700 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:16.217482 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:16.217659 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:16.220139 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "f6c322afff1648789841b8f687850cb0"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "1463fd251a9a9a04130e4d117284fb03"
server_key_iv: "c6cdf9ab39895330edb418a903e18337"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.220788 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f6c322afff1648789841b8f687850cb0"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "1463fd251a9a9a04130e4d117284fb03"
server_key_iv: "c6cdf9ab39895330edb418a903e18337"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.225116 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:36:16.228413 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.230103 10700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:36:16.230250 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "f6c322afff1648789841b8f687850cb0"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "1463fd251a9a9a04130e4d117284fb03"
server_key_iv: "c6cdf9ab39895330edb418a903e18337"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.230379 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:16.244941 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:16.248648 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:16.248868 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:16.249516 10700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:16.250547 10700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:16.250600 10700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.250658 10700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:16.250687 10700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.260123 10700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44367
I20260927 12:36:16.260176 10828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44367 every 8 connection(s)
I20260927 12:36:16.261060 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:36:16.270720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10700
I20260927 12:36:16.270835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:36:16.271099 392 external_mini_cluster.cc:1428] Setting key 3e49d70f30b0b02e3924673b58aed129
Sep 27 12:36:16 dist-test-slave-tzgx krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:36:16.276651 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.262945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36445 (local address 127.0.98.62:37243)
0927 12:36:16.263095 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.263098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.263989 (+ 891us) server_negotiation.cc:407] Connection header received
0927 12:36:16.265371 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.265375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.265438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.265523 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:16.267070 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.267644 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.269107 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.269455 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.269583 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.272453 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:16.272458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:16.272462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:36:16.272520 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:16.274175 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.274811 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.274815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.274816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.274905 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.275372 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.275375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.275376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.275613 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:16.275728 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.276326 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.276480 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 12:36:16.277522 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.263307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37243 (local address 127.0.98.1:36445)
0927 12:36:16.263809 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.263855 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.265125 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.265738 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.265746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.266278 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:16.266877 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.266894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.267831 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.267835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.268874 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.268886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.269676 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.269681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.269877 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.270528 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:16.270554 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:16.272243 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:16.274331 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:36:16.274338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:36:16.274353 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:36:16.274678 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:36:16.274997 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:36:16.275000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:36:16.275002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:36:16.275233 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:36:16.275729 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:16.275737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:16.276039 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.276253 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.276619 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":105,"threads_started":1}
I20260927 12:36:16.278816 10829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37243
I20260927 12:36:16.279102 10829 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:16.279729 10829 heartbeater.cc:511] Master 127.0.98.62:37243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:36:16.281472 10638 ts_manager.cc:194] Registered new tserver with Master: f6c322afff1648789841b8f687850cb0 (127.0.98.1:44367)
I20260927 12:36:16.282845 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:36:16.303205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37243
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:36:16.421576 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:16.421876 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:16.421949 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:16.426498 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:16.426602 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:16.426693 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:36:16.432447 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:16.433631 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:16.434711 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:16.445039 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:16.445201 10834 server_base.cc:1034] running on GCE node
W20260927 12:36:16.445448 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:16.445039 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:16.446198 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:16.447101 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:16.448274 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790512576448246 us; error 48 us; skew 500 ppm
Sep 27 12:36:16 dist-test-slave-tzgx krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:36:16.451035 10834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:36:16.452039 10834 webserver.cc:460] Webserver started at http://127.0.98.2:46515/ using document root <none> and password file <none>
I20260927 12:36:16.452668 10834 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:16.452733 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:16.452931 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:16.455101 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "59e56002621c4cf798e0e0a8ab1a23d1"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "500fb2ba07a314153ad1122205fc7c48"
server_key_iv: "dd8b10eb78da39e5a1bd87d02cd24721"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.455711 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "59e56002621c4cf798e0e0a8ab1a23d1"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "500fb2ba07a314153ad1122205fc7c48"
server_key_iv: "dd8b10eb78da39e5a1bd87d02cd24721"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.460039 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 12:36:16.462771 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.464253 10834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:36:16.464432 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "59e56002621c4cf798e0e0a8ab1a23d1"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "500fb2ba07a314153ad1122205fc7c48"
server_key_iv: "dd8b10eb78da39e5a1bd87d02cd24721"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.464562 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:16.491637 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:16.494904 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:16.495083 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:16.495721 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:16.496618 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:16.496665 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.496706 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:16.496734 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.505815 10834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36789
I20260927 12:36:16.505883 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36789 every 8 connection(s)
I20260927 12:36:16.506767 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:36:16.511538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10834
I20260927 12:36:16.511642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:36:16.511897 392 external_mini_cluster.cc:1428] Setting key 7a2598902d893e3f10fb38082fd65662
Sep 27 12:36:16 dist-test-slave-tzgx krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:36:16.522053 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.508971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38849 (local address 127.0.98.62:37243)
0927 12:36:16.509139 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.509143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.510085 (+ 942us) server_negotiation.cc:407] Connection header received
0927 12:36:16.511361 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.511365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.511423 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.511496 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:16.513018 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.513604 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.514793 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.515029 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.515118 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.517989 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:16.517993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:16.517995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:36:16.518021 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:16.519663 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.520371 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.520374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.520375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.520459 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.520875 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.520878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.520879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.521078 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:16.521166 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.521731 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.521881 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:36:16.522907 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.509381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37243 (local address 127.0.98.2:38849)
0927 12:36:16.509916 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.509962 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.511143 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.511700 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.511708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.512238 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:16.512868 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.512885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.513738 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.513741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.514656 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.514668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.515181 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.515185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.515358 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.516000 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:16.516017 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:16.517784 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:16.519852 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:36:16.519859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:36:16.519877 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:36:16.520259 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:36:16.520550 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:36:16.520553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:36:16.520555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:36:16.520768 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:36:16.521183 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:16.521191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:16.521481 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.521652 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.522010 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:36:16.524773 10963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37243
I20260927 12:36:16.525076 10963 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:16.525879 10963 heartbeater.cc:511] Master 127.0.98.62:37243 requested a full tablet report, sending...
I20260927 12:36:16.527335 10638 ts_manager.cc:194] Registered new tserver with Master: 59e56002621c4cf798e0e0a8ab1a23d1 (127.0.98.2:36789)
I20260927 12:36:16.527973 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:36:16.544884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37243
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:36:16.682907 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:36:16.683212 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:36:16.683259 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:36:16.687934 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:36:16.688027 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:36:16.688118 10968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:36:16.693773 10968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:36:16.694866 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:36:16.695905 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:36:16.707056 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:36:16.707185 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:16.707149 10968 server_base.cc:1034] running on GCE node
W20260927 12:36:16.707067 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:36:16.707887 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:36:16.708725 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:36:16.709944 10968 hybrid_clock.cc:648] HybridClock initialized: now 1790512576709937 us; error 48 us; skew 500 ppm
Sep 27 12:36:16 dist-test-slave-tzgx krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:36:16.712987 10968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:36:16.713976 10968 webserver.cc:460] Webserver started at http://127.0.98.3:45001/ using document root <none> and password file <none>
I20260927 12:36:16.714555 10968 fs_manager.cc:359] Metadata directory not provided
I20260927 12:36:16.714625 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:36:16.714843 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:36:16.717077 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "eca75769657947f9a77d6c7b6a3f456f"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "a24c6442f12555c7a4ab6d5a6a90e9d4"
server_key_iv: "13797bddb0b146b0187aa44a8e31adc5"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.717661 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eca75769657947f9a77d6c7b6a3f456f"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "a24c6442f12555c7a4ab6d5a6a90e9d4"
server_key_iv: "13797bddb0b146b0187aa44a8e31adc5"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.722146 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:36:16.725809 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.727632 10968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 12:36:16.727804 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "eca75769657947f9a77d6c7b6a3f456f"
format_stamp: "Formatted at 2026-09-27 12:36:16 on dist-test-slave-tzgx"
server_key: "a24c6442f12555c7a4ab6d5a6a90e9d4"
server_key_iv: "13797bddb0b146b0187aa44a8e31adc5"
server_key_version: "encryptionkey@0"
I20260927 12:36:16.727941 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:36:16.745337 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:36:16.748898 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:36:16.749137 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:36:16.749861 10968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:36:16.750802 10968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:36:16.750859 10968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.750919 10968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:36:16.750952 10968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:36:16.760442 10968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36089
I20260927 12:36:16.760480 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36089 every 8 connection(s)
I20260927 12:36:16.761404 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:36:16.762740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 10968
I20260927 12:36:16.762863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:36:16.763190 392 external_mini_cluster.cc:1428] Setting key 88664e68db0f7fed8e81477040bac3fe
Sep 27 12:36:16 dist-test-slave-tzgx krb5kdc[10593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:36:16.776847 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.763514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49475 (local address 127.0.98.62:37243)
0927 12:36:16.763665 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.763670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.764781 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 12:36:16.766400 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.766403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.766452 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.766526 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:16.767939 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.768620 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.769946 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.770177 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.770272 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.772919 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:16.772923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:16.772925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:36:16.772954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:16.774383 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.775046 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.775050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.775051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.775146 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.775518 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.775521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.775523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.775727 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:16.775819 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.776563 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.776705 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:36:16.777987 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.764027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37243 (local address 127.0.98.3:49475)
0927 12:36:16.764652 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.764705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.766162 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.766763 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.766772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.767213 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:36:16.767756 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.767771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.768773 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.768776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.769784 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.769796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.770371 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.770375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.770542 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.771189 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:36:16.771207 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:36:16.772711 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:36:16.774512 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:36:16.774518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:36:16.774534 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:36:16.774892 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:36:16.775250 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:36:16.775252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:36:16.775254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:36:16.775409 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:36:16.775946 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:36:16.775955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:36:16.776285 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.776509 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.776899 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":161,"threads_started":1}
I20260927 12:36:16.779271 11097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37243
I20260927 12:36:16.779500 11097 heartbeater.cc:464] Registering TS with master...
I20260927 12:36:16.780033 11097 heartbeater.cc:511] Master 127.0.98.62:37243 requested a full tablet report, sending...
I20260927 12:36:16.781129 10638 ts_manager.cc:194] Registered new tserver with Master: eca75769657947f9a77d6c7b6a3f456f (127.0.98.3:36089)
I20260927 12:36:16.781744 10638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49475
I20260927 12:36:16.790333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:36:16.800518 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.792549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.0.98.62:37243)
0927 12:36:16.792662 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.792666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.792861 (+ 195us) server_negotiation.cc:407] Connection header received
0927 12:36:16.792982 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.792985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.793029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.793093 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:36:16.793881 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.794418 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.795324 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.795569 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.795783 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.797110 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:36:16.797114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:36:16.797116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:36:16.797145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:36:16.798574 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.799074 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.799077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.799078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.799166 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:36:16.799494 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:36:16.799497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:36:16.799498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:36:16.799761 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:36:16.799952 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.800270 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.800389 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:36:16.804162 10638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:36:16.807503 10638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:36:16.822432 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.816533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44367 (local address 127.0.0.1:60240)
0927 12:36:16.816947 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.816999 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.817129 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.817916 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.817923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.817945 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:36:16.818095 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.818103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.819480 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.819484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.820613 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.820623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.821847 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.821851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.821971 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.822011 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.822126 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.822269 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260927 12:36:16.822433 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.816855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.98.1:44367)
0927 12:36:16.817413 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.817420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.817443 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:36:16.817519 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.817524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.817746 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.818240 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:36:16.818366 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.819350 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.820861 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.821731 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.821829 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.822037 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.822166 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.822269 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":107,"threads_started":1}
I20260927 12:36:16.822798 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.817308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36089 (local address 127.0.0.1:57972)
0927 12:36:16.817733 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.817745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.817826 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.818572 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.818575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.818589 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:36:16.818736 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.818743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.820292 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.820296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.821320 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.821329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.822391 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.822395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.822506 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.822534 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.822583 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.822665 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":48,"thread_start_us":67,"threads_started":1}
I20260927 12:36:16.823284 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.816620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36789 (local address 127.0.0.1:39290)
0927 12:36:16.817261 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.817274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.817346 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.818489 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.818492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.818509 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:36:16.818643 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.818650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.820253 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.820257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.821511 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.821522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.822915 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.822919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.823028 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.823049 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.823096 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.823164 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260927 12:36:16.823433 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.817667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.98.3:36089)
0927 12:36:16.818129 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.818140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.818163 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:36:16.818239 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.818244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.818430 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.818678 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:36:16.818840 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.820135 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.821442 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.822245 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.822979 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.823152 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.823203 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.823302 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":172,"threads_started":1}
I20260927 12:36:16.824671 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.817233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.0.98.2:36789)
0927 12:36:16.818044 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.818054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.818077 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:36:16.818148 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.818153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.818334 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.818492 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:36:16.818769 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.820074 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.821651 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.822448 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.823480 (+ 1032us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.824369 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.824425 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.824527 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":115,"threads_started":1}
I20260927 12:36:16.825865 10763 tablet_service.cc:1467] Processing CreateTablet for tablet bd2b969ef01e4518963f164686f9578f (DEFAULT_TABLE table=test-table [id=0c599fe50a3949fe979bbc485b143946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:36:16.826093 11031 tablet_service.cc:1467] Processing CreateTablet for tablet bd2b969ef01e4518963f164686f9578f (DEFAULT_TABLE table=test-table [id=0c599fe50a3949fe979bbc485b143946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:36:16.827081 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2b969ef01e4518963f164686f9578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:36:16.827085 11031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2b969ef01e4518963f164686f9578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:36:16.827097 10897 tablet_service.cc:1467] Processing CreateTablet for tablet bd2b969ef01e4518963f164686f9578f (DEFAULT_TABLE table=test-table [id=0c599fe50a3949fe979bbc485b143946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:36:16.828341 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2b969ef01e4518963f164686f9578f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:36:16.834641 11114 tablet_bootstrap.cc:492] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: Bootstrap starting.
I20260927 12:36:16.835484 11115 tablet_bootstrap.cc:492] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Bootstrap starting.
I20260927 12:36:16.835951 11116 tablet_bootstrap.cc:492] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: Bootstrap starting.
I20260927 12:36:16.837659 11114 tablet_bootstrap.cc:654] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: Neither blocks nor log segments found. Creating new log.
I20260927 12:36:16.838044 11115 tablet_bootstrap.cc:654] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Neither blocks nor log segments found. Creating new log.
I20260927 12:36:16.838138 11116 tablet_bootstrap.cc:654] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: Neither blocks nor log segments found. Creating new log.
I20260927 12:36:16.838886 11116 log.cc:824] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:36:16.838874 11114 log.cc:824] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:36:16.838973 11115 log.cc:824] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:36:16.841059 11116 tablet_bootstrap.cc:492] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: No bootstrap required, opened a new log
I20260927 12:36:16.841275 11116 ts_tablet_manager.cc:1397] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:36:16.841284 11115 tablet_bootstrap.cc:492] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: No bootstrap required, opened a new log
I20260927 12:36:16.841284 11114 tablet_bootstrap.cc:492] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: No bootstrap required, opened a new log
I20260927 12:36:16.841481 11114 ts_tablet_manager.cc:1397] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:36:16.841481 11115 ts_tablet_manager.cc:1397] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:36:16.844480 11116 raft_consensus.cc:348] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.844762 11116 raft_consensus.cc:374] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:36:16.844817 11116 raft_consensus.cc:729] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e56002621c4cf798e0e0a8ab1a23d1, State: Initialized, Role: FOLLOWER
I20260927 12:36:16.845278 11116 consensus_queue.cc:260] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.845582 11115 raft_consensus.cc:348] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.845881 11115 raft_consensus.cc:374] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:36:16.845939 11115 raft_consensus.cc:729] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca75769657947f9a77d6c7b6a3f456f, State: Initialized, Role: FOLLOWER
I20260927 12:36:16.846009 10963 heartbeater.cc:503] Master 127.0.98.62:37243 was elected leader, sending a full tablet report...
I20260927 12:36:16.846371 11116 ts_tablet_manager.cc:1428] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:36:16.846381 11115 consensus_queue.cc:260] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.847039 11097 heartbeater.cc:503] Master 127.0.98.62:37243 was elected leader, sending a full tablet report...
I20260927 12:36:16.847380 11115 ts_tablet_manager.cc:1428] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:36:16.848308 11114 raft_consensus.cc:348] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.848593 11114 raft_consensus.cc:374] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:36:16.848649 11114 raft_consensus.cc:729] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c322afff1648789841b8f687850cb0, State: Initialized, Role: FOLLOWER
I20260927 12:36:16.849066 11114 consensus_queue.cc:260] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.850098 11114 ts_tablet_manager.cc:1428] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260927 12:36:16.850467 10829 heartbeater.cc:503] Master 127.0.98.62:37243 was elected leader, sending a full tablet report...
I20260927 12:36:16.930940 11120 raft_consensus.cc:482] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:36:16.931155 11120 raft_consensus.cc:504] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.932487 11120 leader_election.cc:288] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c322afff1648789841b8f687850cb0 (127.0.98.1:44367), eca75769657947f9a77d6c7b6a3f456f (127.0.98.3:36089)
I20260927 12:36:16.936733 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.932745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44367 (local address 127.0.98.2:37325)
0927 12:36:16.932917 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.932934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.933040 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.933281 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.933286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.933319 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:36:16.933446 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.933454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.934354 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.934357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.935369 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.935378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.936258 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.936261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.936369 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.936395 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.936505 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.936583 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:36:16.936770 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.932849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37325 (local address 127.0.98.1:44367)
0927 12:36:16.932974 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.932979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.932995 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:36:16.933135 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.933138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.933184 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.933256 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:36:16.933547 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.934228 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.935517 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.936150 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.936238 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.936300 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.936564 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.936639 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:36:16.937158 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.933310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42079 (local address 127.0.98.3:36089)
0927 12:36:16.933438 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.933443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.933635 (+ 192us) server_negotiation.cc:407] Connection header received
0927 12:36:16.933735 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.933738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.933778 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.933867 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:36:16.934149 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.934802 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.936022 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.936600 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.936707 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.936763 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.936978 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.937057 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:36:16.937556 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.933193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36089 (local address 127.0.98.2:42079)
0927 12:36:16.933538 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:16.933550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:36:16.933646 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:36:16.933890 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:36:16.933893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:36:16.933907 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:36:16.934021 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.934027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.934924 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.934927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.935893 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:36:16.935903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.936724 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.936728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:36:16.936827 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.936851 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:36:16.937327 (+ 476us) client_negotiation.cc:240] Negotiation successful
0927 12:36:16.937397 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":120,"threads_started":1}
I20260927 12:36:16.937589 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2b969ef01e4518963f164686f9578f" candidate_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c322afff1648789841b8f687850cb0" is_pre_election: true
I20260927 12:36:16.937955 10783 raft_consensus.cc:2454] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e56002621c4cf798e0e0a8ab1a23d1 in term 0.
I20260927 12:36:16.938266 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2b969ef01e4518963f164686f9578f" candidate_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca75769657947f9a77d6c7b6a3f456f" is_pre_election: true
I20260927 12:36:16.938448 10851 leader_election.cc:302] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e56002621c4cf798e0e0a8ab1a23d1, f6c322afff1648789841b8f687850cb0; no voters:
I20260927 12:36:16.938580 11051 raft_consensus.cc:2454] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e56002621c4cf798e0e0a8ab1a23d1 in term 0.
I20260927 12:36:16.938722 11120 raft_consensus.cc:2781] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:36:16.938807 11120 raft_consensus.cc:482] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:36:16.938856 11120 raft_consensus.cc:3037] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:36:16.940001 11120 raft_consensus.cc:504] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.940375 11120 leader_election.cc:288] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [CANDIDATE]: Term 1 election: Requested vote from peers f6c322afff1648789841b8f687850cb0 (127.0.98.1:44367), eca75769657947f9a77d6c7b6a3f456f (127.0.98.3:36089)
I20260927 12:36:16.940708 10783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2b969ef01e4518963f164686f9578f" candidate_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c322afff1648789841b8f687850cb0"
I20260927 12:36:16.940759 11051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2b969ef01e4518963f164686f9578f" candidate_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca75769657947f9a77d6c7b6a3f456f"
I20260927 12:36:16.940836 10783 raft_consensus.cc:3037] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:36:16.940886 11051 raft_consensus.cc:3037] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:36:16.941872 10783 raft_consensus.cc:2454] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e56002621c4cf798e0e0a8ab1a23d1 in term 1.
I20260927 12:36:16.941944 11051 raft_consensus.cc:2454] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e56002621c4cf798e0e0a8ab1a23d1 in term 1.
I20260927 12:36:16.942150 10851 leader_election.cc:302] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e56002621c4cf798e0e0a8ab1a23d1, f6c322afff1648789841b8f687850cb0; no voters:
I20260927 12:36:16.942335 11120 raft_consensus.cc:2781] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:36:16.942559 11120 raft_consensus.cc:686] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 1 LEADER]: Becoming Leader. State: Replica: 59e56002621c4cf798e0e0a8ab1a23d1, State: Running, Role: LEADER
I20260927 12:36:16.942919 11120 consensus_queue.cc:237] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } }
I20260927 12:36:16.946216 10637 catalog_manager.cc:5495] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e56002621c4cf798e0e0a8ab1a23d1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e56002621c4cf798e0e0a8ab1a23d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 } health_report { overall_health: UNKNOWN } } }
I20260927 12:36:16.978124 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:16.974692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.0.98.2:36789)
0927 12:36:16.974810 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:36:16.974813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:36:16.974827 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:36:16.974927 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:36:16.974929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:36:16.974985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:36:16.975068 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:36:16.975283 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.975797 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.976708 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:36:16.976954 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:36:16.977049 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:36:16.977299 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:36:16.977759 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:36:16.977847 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:36:16.977935 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 12:36:16.977993 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:36:16.984490 11051 raft_consensus.cc:1261] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 1 FOLLOWER]: Refusing update from remote peer 59e56002621c4cf798e0e0a8ab1a23d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:36:16.984534 10783 raft_consensus.cc:1261] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 1 FOLLOWER]: Refusing update from remote peer 59e56002621c4cf798e0e0a8ab1a23d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:36:16.985247 11120 consensus_queue.cc:1035] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c322afff1648789841b8f687850cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:36:16.985492 11124 consensus_queue.cc:1035] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca75769657947f9a77d6c7b6a3f456f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:36:16.993484 11129 mvcc.cc:205] Tried to move back new op lower bound from 7333939515320238080 to 7333939515160723456. Current Snapshot: MvccSnapshot[applied={T|T < 7333939515320238080}]
I20260927 12:36:16.994920 11128 mvcc.cc:205] Tried to move back new op lower bound from 7333939515320238080 to 7333939515160723456. Current Snapshot: MvccSnapshot[applied={T|T < 7333939515320238080}]
I20260927 12:36:17.001770 10638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:36:17.001999 10638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:36:17.004755 10638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 320256de75bc4732a8655369e99228a4: Sending DeleteTablet for 3 replicas of tablet bd2b969ef01e4518963f164686f9578f
I20260927 12:36:17.005606 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2b969ef01e4518963f164686f9578f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:36:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260927 12:36:17.005640 11031 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2b969ef01e4518963f164686f9578f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:36:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57972
I20260927 12:36:17.005765 10763 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2b969ef01e4518963f164686f9578f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:36:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60240
I20260927 12:36:17.006153 11136 tablet_replica.cc:330] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: stopping tablet replica
I20260927 12:36:17.006341 11138 tablet_replica.cc:330] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0: stopping tablet replica
I20260927 12:36:17.006716 11137 tablet_replica.cc:330] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: stopping tablet replica
I20260927 12:36:17.006707 11138 raft_consensus.cc:2229] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:36:17.007010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10700
I20260927 12:36:17.007032 11137 raft_consensus.cc:2229] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:36:17.007282 11138 raft_consensus.cc:2258] T bd2b969ef01e4518963f164686f9578f P f6c322afff1648789841b8f687850cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:36:17.007345 11137 raft_consensus.cc:2258] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:36:17.009969 11136 raft_consensus.cc:2229] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:36:17.010458 11136 raft_consensus.cc:2258] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:36:17.011546 11137 ts_tablet_manager.cc:1905] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:36:17.012040 11136 ts_tablet_manager.cc:1905] T bd2b969ef01e4518963f164686f9578f P 59e56002621c4cf798e0e0a8ab1a23d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:36:17.014317 10623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44367 (error 108)
I20260927 12:36:17.014515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10834
I20260927 12:36:17.015921 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:17.015530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44367 (local address 127.0.0.1:60246)
0927 12:36:17.015747 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:17.015825 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 12:36:17.016076 11137 ts_tablet_manager.cc:1918] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 12:36:17.016124 10623 catalog_manager.cc:4584] TS f6c322afff1648789841b8f687850cb0 (127.0.98.1:44367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd2b969ef01e4518963f164686f9578f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44367: connect: Connection refused (error 111)
I20260927 12:36:17.016213 11137 log.cc:1196] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/bd2b969ef01e4518963f164686f9578f
I20260927 12:36:17.016492 11137 ts_tablet_manager.cc:1939] T bd2b969ef01e4518963f164686f9578f P eca75769657947f9a77d6c7b6a3f456f: Deleting consensus metadata
I20260927 12:36:17.017702 10623 catalog_manager.cc:4841] TS eca75769657947f9a77d6c7b6a3f456f (127.0.98.3:36089): tablet bd2b969ef01e4518963f164686f9578f (table test-table [id=0c599fe50a3949fe979bbc485b143946]) successfully deleted
I20260927 12:36:17.020082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10968
I20260927 12:36:17.020534 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:36:17.020231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36789 (local address 127.0.0.1:39310)
0927 12:36:17.020402 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 12:36:17.020452 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 12:36:17.025223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 10606
2026-09-27T12:36:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2360 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:36:17 dist-test-slave-tzgx krb5kdc[11142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:36:17 dist-test-slave-tzgx krb5kdc[11142](info): set up 2 sockets
Sep 27 12:36:17 dist-test-slave-tzgx krb5kdc[11142](info): commencing operation
krb5kdc: starting...
W20260927 12:36:18.021072 392 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:36:18 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:36:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 12:36:18.212349 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRFp9Fb/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/postgres -l logfile start
2026-09-27 12:36:21.289 UTC [11172] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 12:36:21.289 UTC [11172] LOG: listening on IPv4 address "127.0.98.20", port 53823
2026-09-27 12:36:21.294 UTC [11172] LOG: listening on Unix socket "/tmp/.s.PGSQL.53823"
2026-09-27 12:36:21.316 UTC [11175] LOG: database system was shut down at 2026-09-27 12:36:19 UTC
2026-09-27 12:36:21.319 UTC [11172] LOG: database system is ready to accept connections
I20260927 12:36:22.220839 392 mini_postgres.cc:96] Postgres bound to 53823
2026-09-27 12:36:22.228 UTC [11183] FATAL: database "slave" does not exist
127.0.98.20:53823 - accepting connections
I20260927 12:36:22.229071 392 mini_ranger.cc:161] Starting Ranger...
I20260927 12:36:22.249392 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 12:36:23.005360 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 12:36:23.005486 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-admin
2026-09-27 12:36:23,286 [I] DB FLAVOR :POSTGRES
2026-09-27 12:36:23,287 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:36:23,287 [I] Checking connection..
2026-09-27 12:36:23,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 12:36:23,539 [I] Checking connection passed.
2026-09-27 12:36:23,539 [I] --------- Verifying version history table ---------
2026-09-27 12:36:23,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:36:23,820 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:36:23,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:36:24,082 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:36:24,082 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 12:36:24,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 12:36:24.324 UTC [11247] WARNING: there is no transaction in progress
2026-09-27 12:36:24,330 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 12:36:24,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:36:24,587 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 12:36:24,588 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:36:24,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 12:36:24,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 12:36:25,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tzgx.c.gcp-upstream.internal','N') ;"
2026-09-27 12:36:25,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:36:25,612 [I] Table x_portal_user does not exist in database ranger
2026-09-27 12:36:25,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:36:25,850 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 12:36:25,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:36:26,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:36:26,378 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 12:36:26,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 12:36:27.471 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.489 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.508 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.527 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.549 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.572 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.644 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.653 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.670 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.681 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.692 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.708 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.718 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.725 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.733 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:27.741 UTC [11355] WARNING: there is no transaction in progress
2026-09-27 12:36:28,524 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 12:36:28,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 12:36:28,771 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 12:36:28,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:36:29,019 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 12:36:29,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:36:29,279 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 12:36:29,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:36:29,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:36:29,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tzgx.c.gcp-upstream.internal';"
2026-09-27 12:36:30,054 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 12:36:30,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 12:36:30,305 [I] DB_PATCHES have already been applied
I20260927 12:36:30.311539 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRFp9Fb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 12:36:30.311623 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 12:36:30.316020 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54677
Sep 27, 2026 12:36:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:36:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:36:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:36:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:36:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:36:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:36:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 12:36:31.355459 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.126s user 0.006s sys 0.007s
Sep 27, 2026 12:36:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:36:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 12:36:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:36:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:36:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 12:36:54.354144 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 12:36:54 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512614, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 12:36:54.445813 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 12:36:54.459801 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 12:36:55.585774 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 12:36:55.585855 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms
I20260927 12:36:55.585945 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53823
2026-09-27 12:36:55,768 [I] DB FLAVOR :POSTGRES
2026-09-27 12:36:55,768 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:36:55,768 [I] Checking connection
2026-09-27 12:36:55,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 12:36:56,018 [I] connection success
2026-09-27 12:36:56,018 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 12:36:56,018 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 12:36:56,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 12:36:56,301 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 12:36:56,301 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:36:56,301 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 12:36:56,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53823/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 12:36:56,620 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 12:36:57.117286 392 mini_ranger_kms.cc:310] Created kms service
I20260927 12:36:57.334060 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 12:36:57.377743 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 12:36:57 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512614, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:36:57.772346 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 12:36:57.772495 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 12:36:57.772512 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 12:36:57.776503 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11536
I20260927 12:36:57.776631 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58623
Sep 27, 2026 12:36:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:36:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:36:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:36:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:36:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:36:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 12:36:59.846062 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.400s user 0.015s sys 0.150s
I20260927 12:36:59.846174 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 12:36:59.846283 392 mini_ranger_kms.cc:401] 127.0.98.20:58623/kms/v1/keys
Sep 27 12:37:00 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512620, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 12:37:00 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512620, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:37:00 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512620, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:37:03 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:37:03.408003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37277
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37277
--ranger_config_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58623/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:03.520905 11586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:03.521193 11586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:03.521231 11586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:03.525405 11586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:37:03.525470 11586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:03.525489 11586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:03.525504 11586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:37:03.525518 11586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:37:03.531400 11586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58623/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37277
--ranger_config_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:03.532521 11586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:03.533560 11586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:03.542531 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:03.542646 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:03.542557 11591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:03.542713 11586 server_base.cc:1034] running on GCE node
I20260927 12:37:03.543694 11586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:03.544947 11586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:03.546132 11586 hybrid_clock.cc:648] HybridClock initialized: now 1790512623546119 us; error 31 us; skew 500 ppm
Sep 27 12:37:03 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:03.552465 11586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:37:03.553529 11586 webserver.cc:460] Webserver started at http://127.0.98.62:43757/ using document root <none> and password file <none>
I20260927 12:37:03.554090 11586 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:03.554152 11586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:03.554347 11586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:03 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:03.610472 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "b6d20ee0b0a2499ca380693ecb1aeb23"
format_stamp: "Formatted at 2026-09-27 12:37:03 on dist-test-slave-tzgx"
server_key: "26530acb317be5ec134318293923273e"
server_key_iv: "1c499294bdcd7c6221a662efd5951589"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:03.611090 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6d20ee0b0a2499ca380693ecb1aeb23"
format_stamp: "Formatted at 2026-09-27 12:37:03 on dist-test-slave-tzgx"
server_key: "26530acb317be5ec134318293923273e"
server_key_iv: "1c499294bdcd7c6221a662efd5951589"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:03.614959 11586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:37:03.632485 11601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:03.633985 11586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:37:03.634124 11586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "b6d20ee0b0a2499ca380693ecb1aeb23"
format_stamp: "Formatted at 2026-09-27 12:37:03 on dist-test-slave-tzgx"
server_key: "26530acb317be5ec134318293923273e"
server_key_iv: "1c499294bdcd7c6221a662efd5951589"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:03.634263 11586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:03.649444 11586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:03.652318 11586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:03.652518 11586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:03.660000 11586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37277
I20260927 12:37:03.660032 11653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37277 every 8 connection(s)
I20260927 12:37:03.660990 11586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:37:03.664318 11654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:03.665737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 11586
I20260927 12:37:03.665849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:37:03.666141 392 external_mini_cluster.cc:1428] Setting key 0c7920e11b51cfc63969320313090d14
I20260927 12:37:03.671952 11654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23: Bootstrap starting.
Sep 27 12:37:03 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:03.674784 11654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:03.675701 11654 log.cc:824] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:03.678128 11654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23: No bootstrap required, opened a new log
I20260927 12:37:03.680658 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:03.667591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.0.98.62:37277)
0927 12:37:03.667986 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:03.668002 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:03.668037 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:37:03.668756 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:03.668776 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:03.669116 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:03.669446 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:03.670218 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:03.671190 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:03.672016 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:03.672559 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:03.672716 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:03.675052 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:03.675058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:03.675073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:03.675103 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:03.677351 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:03.677897 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:03.677903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:03.677909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:03.678041 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:03.678335 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:03.678339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:03.678341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:03.678874 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:03.679057 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:03.679360 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 12:37:03.679643 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260927 12:37:03.682219 11654 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260927 12:37:03.682500 11654 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:03.682590 11654 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d20ee0b0a2499ca380693ecb1aeb23, State: Initialized, Role: FOLLOWER
I20260927 12:37:03.683115 11654 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260927 12:37:03.683282 11654 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:37:03.683367 11654 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:37:03.683517 11654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:03.684549 11654 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260927 12:37:03.684908 11654 leader_election.cc:302] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6d20ee0b0a2499ca380693ecb1aeb23; no voters:
I20260927 12:37:03.685400 11654 leader_election.cc:288] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:37:03.685478 11659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:37:03.685724 11659 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [term 1 LEADER]: Becoming Leader. State: Replica: b6d20ee0b0a2499ca380693ecb1aeb23, State: Running, Role: LEADER
I20260927 12:37:03.686092 11659 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } }
I20260927 12:37:03.686774 11654 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:37:03.687734 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d20ee0b0a2499ca380693ecb1aeb23. Latest consensus state: current_term: 1 leader_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } } }
I20260927 12:37:03.687862 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [sys.catalog]: This master's current role is: LEADER
I20260927 12:37:03.688185 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d20ee0b0a2499ca380693ecb1aeb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37277 } } }
I20260927 12:37:03.688263 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23 [sys.catalog]: This master's current role is: LEADER
I20260927 12:37:03.689050 11654 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:37:03.689818 11668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:37:03.695683 11668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:37:03.705688 11668 catalog_manager.cc:1347] Generated new cluster ID: 4b02b0cff9954e92ada0381b6e48d847
I20260927 12:37:03.705773 11668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:37:03.719521 11668 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:37:03.721033 11668 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:37:03.751540 11668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23: Generated new TSK 0
I20260927 12:37:03.752804 11668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:37:04.485020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:04.624931 11699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:04.625207 11699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:04.625252 11699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:04.630028 11699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:04.630117 11699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:04.630205 11699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:37:04.636644 11699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:04.638019 11699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:04.639230 11699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:04.650996 11704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:04.651023 11707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:04.651429 11699 server_base.cc:1034] running on GCE node
W20260927 12:37:04.651078 11705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:04.652118 11699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:04.652842 11699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:04.654026 11699 hybrid_clock.cc:648] HybridClock initialized: now 1790512624654012 us; error 35 us; skew 500 ppm
Sep 27 12:37:04 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:04.656947 11699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:37:04.657899 11699 webserver.cc:460] Webserver started at http://127.0.98.1:40455/ using document root <none> and password file <none>
I20260927 12:37:04.658449 11699 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:04.658512 11699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:04.658720 11699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:04 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:04.675756 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "186b54f598ba4ee59cb08e6df2065572"
format_stamp: "Formatted at 2026-09-27 12:37:04 on dist-test-slave-tzgx"
server_key: "29b01f77716f2c4bb3e9ed07a3730c26"
server_key_iv: "1c206cce5d40dcd1b2b46fd1e3882ff0"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:04.676585 11699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "186b54f598ba4ee59cb08e6df2065572"
format_stamp: "Formatted at 2026-09-27 12:37:04 on dist-test-slave-tzgx"
server_key: "29b01f77716f2c4bb3e9ed07a3730c26"
server_key_iv: "1c206cce5d40dcd1b2b46fd1e3882ff0"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:04.682125 11699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 12:37:04.704257 11716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:04.706054 11699 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260927 12:37:04.706239 11699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "186b54f598ba4ee59cb08e6df2065572"
format_stamp: "Formatted at 2026-09-27 12:37:04 on dist-test-slave-tzgx"
server_key: "29b01f77716f2c4bb3e9ed07a3730c26"
server_key_iv: "1c206cce5d40dcd1b2b46fd1e3882ff0"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:04.706399 11699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:04.740501 11699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:04.744421 11699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:04.744618 11699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:04.745487 11699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:04.746613 11699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:04.746680 11699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:04.746733 11699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:04.746773 11699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:04.759666 11699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34367
I20260927 12:37:04.759717 11829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34367 every 8 connection(s)
I20260927 12:37:04.760758 11699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:37:04.770725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 11699
I20260927 12:37:04.770854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:37:04.771153 392 external_mini_cluster.cc:1428] Setting key 039a355d5b45066199c3c72d8959260c
Sep 27 12:37:04 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:04.781158 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:04.765267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46235 (local address 127.0.98.62:37277)
0927 12:37:04.767377 (+ 2110us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:04.767383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:04.767398 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:37:04.767443 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:04.767446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:04.767501 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:04.767594 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:04.769232 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:04.770108 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:04.771905 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:04.772285 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:04.772379 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:04.776135 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:04.776140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:04.776143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:04.776179 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:04.778348 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:04.779123 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:04.779127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:04.779129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:04.779228 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:04.779648 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:04.779652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:04.779654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:04.779906 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:04.779981 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:04.780796 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 12:37:04.780951 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":93,"threads_started":1}
I20260927 12:37:04.783356 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:04.764061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37277 (local address 127.0.98.1:46235)
0927 12:37:04.765739 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:04.765801 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:04.766893 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:04.767918 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:04.767927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:04.768327 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:04.769037 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:04.769059 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:04.770271 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:04.770274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:04.771753 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:04.771766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:04.772596 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:04.772600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:04.772812 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:04.773511 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:04.773534 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:04.775855 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:04.778525 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:04.778534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:04.778565 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:04.778985 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:04.779341 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:04.779344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:04.779346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:04.779541 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:04.780064 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:04.780072 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:04.780487 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 12:37:04.781800 (+ 1313us) client_negotiation.cc:240] Negotiation successful
0927 12:37:04.782220 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"thread_start_us":154,"threads_started":1}
I20260927 12:37:04.785415 11830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37277
I20260927 12:37:04.785679 11830 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:04.786293 11830 heartbeater.cc:511] Master 127.0.98.62:37277 requested a full tablet report, sending...
I20260927 12:37:04.788064 11618 ts_manager.cc:194] Registered new tserver with Master: 186b54f598ba4ee59cb08e6df2065572 (127.0.98.1:34367)
I20260927 12:37:04.789510 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46235
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:37:04.820066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:37:04 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 12:37:04.981014 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:04.981449 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:04.981644 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:04.988448 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:04.988543 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:04.988641 11838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:37:04.998062 11838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:04.999856 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:05.001492 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:05.017329 11845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:05.017892 11838 server_base.cc:1034] running on GCE node
W20260927 12:37:05.018205 11847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:05.018828 11844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:05.019402 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:05.020336 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:05.021552 11838 hybrid_clock.cc:648] HybridClock initialized: now 1790512625021520 us; error 52 us; skew 500 ppm
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:05.025574 11838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:37:05.026867 11838 webserver.cc:460] Webserver started at http://127.0.98.2:37767/ using document root <none> and password file <none>
I20260927 12:37:05.027669 11838 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:05.027931 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:05.028548 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:05.057451 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "e1707ff37da44e1898373c325df053e7"
format_stamp: "Formatted at 2026-09-27 12:37:05 on dist-test-slave-tzgx"
server_key: "1d1aeec4177575c2606cd95cba674f89"
server_key_iv: "e8f84c66e1dbeea9d8577b536c15cd57"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:05.058482 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e1707ff37da44e1898373c325df053e7"
format_stamp: "Formatted at 2026-09-27 12:37:05 on dist-test-slave-tzgx"
server_key: "1d1aeec4177575c2606cd95cba674f89"
server_key_iv: "e8f84c66e1dbeea9d8577b536c15cd57"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:05.064450 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 12:37:05.089121 11854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:05.090931 11838 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 12:37:05.091084 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "e1707ff37da44e1898373c325df053e7"
format_stamp: "Formatted at 2026-09-27 12:37:05 on dist-test-slave-tzgx"
server_key: "1d1aeec4177575c2606cd95cba674f89"
server_key_iv: "e8f84c66e1dbeea9d8577b536c15cd57"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:05.091225 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:05.124590 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:05.128540 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:05.128813 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:05.129554 11838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:05.130656 11838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:05.130754 11838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:05.130851 11838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:05.130909 11838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:05.142901 11838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45887
I20260927 12:37:05.143687 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45887 every 8 connection(s)
I20260927 12:37:05.144157 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:37:05.150990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 11838
I20260927 12:37:05.151124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:37:05.151428 392 external_mini_cluster.cc:1428] Setting key 3730c4ee3d5f5fe84a46f376904d65a3
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:05.162770 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.148208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58007 (local address 127.0.98.62:37277)
0927 12:37:05.148374 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.148379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.148707 (+ 328us) server_negotiation.cc:407] Connection header received
0927 12:37:05.149995 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.149999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.150061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.150112 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:05.152010 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.152878 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.154548 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.154862 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.154934 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.158244 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:05.158249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:05.158252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:05.158285 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:05.160041 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:05.160823 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:05.160827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:05.160829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:05.160922 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:05.161351 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:05.161354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:05.161356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:05.161586 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:05.161653 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.162417 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.162568 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:37:05.164824 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.147058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37277 (local address 127.0.98.2:58007)
0927 12:37:05.148539 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.148598 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.149725 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.150545 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.150554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.150916 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:05.151828 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.151851 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.153042 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.153045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.154394 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.154406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.155137 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.155140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.155387 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.156151 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:05.156173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:05.157988 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:05.160254 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:05.160262 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:05.160283 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:05.160691 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:05.161033 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:05.161036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:05.161038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:05.161246 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:05.161734 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:05.161740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:05.162115 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.163268 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.163663 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":144,"threads_started":1}
I20260927 12:37:05.166606 11968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37277
I20260927 12:37:05.166885 11968 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:05.167528 11968 heartbeater.cc:511] Master 127.0.98.62:37277 requested a full tablet report, sending...
I20260927 12:37:05.168735 11618 ts_manager.cc:194] Registered new tserver with Master: e1707ff37da44e1898373c325df053e7 (127.0.98.2:45887)
I20260927 12:37:05.169315 11618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58007
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:37:05.229286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58623/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:05.406536 11973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:05.406957 11973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:05.407034 11973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:05.413712 11973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:05.413800 11973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:05.413888 11973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:37:05.422837 11973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58623/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:05.424360 11973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:05.425751 11973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:05.439790 11979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:05.440390 11973 server_base.cc:1034] running on GCE node
W20260927 12:37:05.440402 11981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 12:37:05.444665 11978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:05.445346 11973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:05.446257 11973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:05.447499 11973 hybrid_clock.cc:648] HybridClock initialized: now 1790512625447432 us; error 87 us; skew 500 ppm
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:05.451339 11973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:37:05.452557 11973 webserver.cc:460] Webserver started at http://127.0.98.3:41289/ using document root <none> and password file <none>
I20260927 12:37:05.453264 11973 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:05.453336 11973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:05.453552 11973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:05.477892 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1"
format_stamp: "Formatted at 2026-09-27 12:37:05 on dist-test-slave-tzgx"
server_key: "9446c7b991284ccc4a894d94bc347b80"
server_key_iv: "8cca8e6099329ae91440cc8ef56f500d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:05.479092 11973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1"
format_stamp: "Formatted at 2026-09-27 12:37:05 on dist-test-slave-tzgx"
server_key: "9446c7b991284ccc4a894d94bc347b80"
server_key_iv: "8cca8e6099329ae91440cc8ef56f500d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:05.485219 11973 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 12:37:05.508340 11988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:05.510058 11973 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 12:37:05.510214 11973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1"
format_stamp: "Formatted at 2026-09-27 12:37:05 on dist-test-slave-tzgx"
server_key: "9446c7b991284ccc4a894d94bc347b80"
server_key_iv: "8cca8e6099329ae91440cc8ef56f500d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:05.510351 11973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:05.539791 11973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:05.543555 11973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:05.543826 11973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:05.544554 11973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:05.545753 11973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:05.545814 11973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:05.545866 11973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:05.545892 11973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:05.557282 11973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39591
I20260927 12:37:05.558477 11973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:37:05.559374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 11973
I20260927 12:37:05.559505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:37:05.559841 392 external_mini_cluster.cc:1428] Setting key be6ced93bb0266e660a367be961e51aa
I20260927 12:37:05.562929 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39591 every 8 connection(s)
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:05.582394 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.566980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48649 (local address 127.0.98.62:37277)
0927 12:37:05.567105 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.567110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.567598 (+ 488us) server_negotiation.cc:407] Connection header received
0927 12:37:05.568993 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.568997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.569054 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.569104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:05.570764 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.571688 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.573439 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.573790 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.573865 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.577276 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:05.577281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:05.577284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:05.577317 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:05.579075 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:05.580187 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:05.580192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:05.580194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:05.580307 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:05.580816 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:05.580820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:05.580822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:05.581071 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:05.581161 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.581977 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.582215 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:37:05.584245 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.565798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37277 (local address 127.0.98.3:48649)
0927 12:37:05.567416 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.567479 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.568725 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.569528 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.569537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.569902 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:05.570570 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.570588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.571908 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.571911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.573309 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.573320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.574086 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.574090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.574318 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.575112 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:05.575136 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:05.577004 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:05.579262 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:05.579269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:05.579291 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:05.579986 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:05.580434 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:05.580437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:05.580439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:05.580673 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:05.581237 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:05.581244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:05.581656 (+ 412us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.582667 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.583080 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"thread_start_us":169,"threads_started":1}
I20260927 12:37:05.585647 12102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37277
I20260927 12:37:05.585947 12102 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:05.586603 12102 heartbeater.cc:511] Master 127.0.98.62:37277 requested a full tablet report, sending...
I20260927 12:37:05.587805 11617 ts_manager.cc:194] Registered new tserver with Master: 68780bb2d16d46cf9f8a7c7e6bf0d9d1 (127.0.98.3:39591)
I20260927 12:37:05.588616 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48649
I20260927 12:37:05.598716 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:37:05.611977 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.601510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.62:37277)
0927 12:37:05.601631 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.601636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.601650 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:37:05.601696 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.601700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.601749 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.601836 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:05.602779 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.603633 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.604972 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.605342 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.605549 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.607264 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:05.607269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:05.607272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:05.607306 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:05.609338 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:05.609986 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:05.609991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:05.609993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:05.610103 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:05.610408 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:05.610412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:05.610414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:05.610810 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:05.610963 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.611254 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.611804 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:37:05.616230 11617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:37:05.621095 11617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:37:05.642916 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.632056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45887 (local address 127.0.0.1:51852)
0927 12:37:05.634050 (+ 1994us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.634111 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.634255 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.637998 (+ 3743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.638043 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.638067 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.638246 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.638254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.639380 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.639384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.641015 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.641026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.642352 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.642356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.642483 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.642524 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.642630 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.642755 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1810,"thread_start_us":155,"threads_started":1}
I20260927 12:37:05.643524 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.632901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.0.98.2:45887)
0927 12:37:05.637428 (+ 4527us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.637444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.637471 (+ 27us) server_negotiation.cc:407] Connection header received
0927 12:37:05.637560 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.637566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.637828 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.637987 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.638366 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.639247 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.641155 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.641979 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.642074 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.642284 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.643276 (+ 992us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.643387 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4150,"thread_start_us":131,"threads_started":1}
I20260927 12:37:05.646768 11902 tablet_service.cc:1467] Processing CreateTablet for tablet 149b727e7cd8441397aefa5be558b970 (DEFAULT_TABLE table=test-table [id=6bed3715e2bd4fcd97d8ed15b27ed2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:37:05.647967 11902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b727e7cd8441397aefa5be558b970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:05.654768 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.633013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34367 (local address 127.0.0.1:40480)
0927 12:37:05.636388 (+ 3375us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.636407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.636501 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.641335 (+ 4834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.641339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.641355 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.641470 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.641476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.643028 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.643032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.644263 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.644273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.654298 (+ 10025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.654302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.654421 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.654454 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.654511 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.654602 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3237,"mutex_wait_us":18,"thread_start_us":82,"threads_started":1}
I20260927 12:37:05.655866 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.633232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.0.98.1:34367)
0927 12:37:05.640650 (+ 7418us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.640658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.640680 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:37:05.640761 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.640766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.641025 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.641632 (+ 607us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.641773 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.642890 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.644399 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.654112 (+ 9713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.655365 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.655550 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.655610 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.655712 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7304,"thread_start_us":98,"threads_started":1}
I20260927 12:37:05.657860 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.633503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39591 (local address 127.0.0.1:57526)
0927 12:37:05.640287 (+ 6784us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.640303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.641192 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.641840 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.641843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.641861 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.641952 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.641958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.643554 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.643557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.653251 (+ 9694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.653273 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.657438 (+ 4165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.657442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.657538 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.657563 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.657630 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.657712 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6700,"mutex_wait_us":28,"thread_start_us":80,"threads_started":1}
I20260927 12:37:05.658279 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.634029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.3:39591)
0927 12:37:05.636805 (+ 2776us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.636823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.640435 (+ 3612us) server_negotiation.cc:407] Connection header received
0927 12:37:05.641340 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.641346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.641539 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.641695 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.642093 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.643427 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.656292 (+ 12865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.657123 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.657227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.657376 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.658038 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.658147 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2608,"thread_start_us":79,"threads_started":1}
I20260927 12:37:05.659451 11763 tablet_service.cc:1467] Processing CreateTablet for tablet 149b727e7cd8441397aefa5be558b970 (DEFAULT_TABLE table=test-table [id=6bed3715e2bd4fcd97d8ed15b27ed2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:37:05.660815 11763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b727e7cd8441397aefa5be558b970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:05.661506 12036 tablet_service.cc:1467] Processing CreateTablet for tablet 149b727e7cd8441397aefa5be558b970 (DEFAULT_TABLE table=test-table [id=6bed3715e2bd4fcd97d8ed15b27ed2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:37:05.668570 12120 tablet_bootstrap.cc:492] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: Bootstrap starting.
I20260927 12:37:05.669909 12036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149b727e7cd8441397aefa5be558b970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:05.670320 12121 tablet_bootstrap.cc:492] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: Bootstrap starting.
I20260927 12:37:05.671706 12120 tablet_bootstrap.cc:654] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:05.672745 12121 tablet_bootstrap.cc:654] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:05.673044 12120 log.cc:824] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:05.673705 12121 log.cc:824] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:05.676321 12120 tablet_bootstrap.cc:492] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: No bootstrap required, opened a new log
I20260927 12:37:05.677824 12121 tablet_bootstrap.cc:492] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: No bootstrap required, opened a new log
I20260927 12:37:05.678081 12121 ts_tablet_manager.cc:1397] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 12:37:05.683009 12121 raft_consensus.cc:348] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.683522 12121 raft_consensus.cc:374] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:05.683714 12121 raft_consensus.cc:729] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186b54f598ba4ee59cb08e6df2065572, State: Initialized, Role: FOLLOWER
I20260927 12:37:05.684405 12121 consensus_queue.cc:260] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.685765 12121 ts_tablet_manager.cc:1428] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 12:37:05.686146 12120 ts_tablet_manager.cc:1397] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260927 12:37:05.690600 12120 raft_consensus.cc:348] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.690979 12120 raft_consensus.cc:374] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:05.691052 12120 raft_consensus.cc:729] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1707ff37da44e1898373c325df053e7, State: Initialized, Role: FOLLOWER
I20260927 12:37:05.691598 12120 consensus_queue.cc:260] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.692533 11830 heartbeater.cc:503] Master 127.0.98.62:37277 was elected leader, sending a full tablet report...
I20260927 12:37:05.692557 12125 tablet_bootstrap.cc:492] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Bootstrap starting.
I20260927 12:37:05.695086 12120 ts_tablet_manager.cc:1428] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 12:37:05.695783 11968 heartbeater.cc:503] Master 127.0.98.62:37277 was elected leader, sending a full tablet report...
I20260927 12:37:05.698768 12125 tablet_bootstrap.cc:654] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:05.699867 12125 log.cc:824] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:05.702350 12125 tablet_bootstrap.cc:492] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: No bootstrap required, opened a new log
I20260927 12:37:05.702576 12125 ts_tablet_manager.cc:1397] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 12:37:05.707073 12125 raft_consensus.cc:348] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.707432 12125 raft_consensus.cc:374] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:05.707491 12125 raft_consensus.cc:729] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68780bb2d16d46cf9f8a7c7e6bf0d9d1, State: Initialized, Role: FOLLOWER
I20260927 12:37:05.708060 12125 consensus_queue.cc:260] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.709116 12102 heartbeater.cc:503] Master 127.0.98.62:37277 was elected leader, sending a full tablet report...
I20260927 12:37:05.709486 12125 ts_tablet_manager.cc:1428] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260927 12:37:05.762833 11831 tablet.cc:2369] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:37:05 dist-test-slave-tzgx krb5kdc[11142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 12:37:05.814970 12103 tablet.cc:2369] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:37:05.862898 12124 raft_consensus.cc:482] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:37:05.863152 12124 raft_consensus.cc:504] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.864774 12124 leader_election.cc:288] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1707ff37da44e1898373c325df053e7 (127.0.98.2:45887), 68780bb2d16d46cf9f8a7c7e6bf0d9d1 (127.0.98.3:39591)
I20260927 12:37:05.871027 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.866031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35919 (local address 127.0.98.2:45887)
0927 12:37:05.866159 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.866163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.866333 (+ 170us) server_negotiation.cc:407] Connection header received
0927 12:37:05.866546 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.866549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.866594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.866639 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.867022 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.867830 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.869637 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.870261 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.870365 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.870445 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.870787 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.870878 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:37:05.871737 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.865181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45887 (local address 127.0.98.1:35919)
0927 12:37:05.866234 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.866250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.866419 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.866779 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.866782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.866799 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.866903 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.866909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.867990 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.867993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.869480 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.869493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.870549 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.870552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.870649 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.870673 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.871482 (+ 809us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.871579 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":131,"threads_started":1}
I20260927 12:37:05.872637 11922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b727e7cd8441397aefa5be558b970" candidate_uuid: "186b54f598ba4ee59cb08e6df2065572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1707ff37da44e1898373c325df053e7" is_pre_election: true
I20260927 12:37:05.872987 11922 raft_consensus.cc:2454] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b54f598ba4ee59cb08e6df2065572 in term 0.
I20260927 12:37:05.873531 11719 leader_election.cc:302] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b54f598ba4ee59cb08e6df2065572, e1707ff37da44e1898373c325df053e7; no voters:
I20260927 12:37:05.873870 12124 raft_consensus.cc:2781] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:37:05.873945 12124 raft_consensus.cc:482] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:37:05.873981 12124 raft_consensus.cc:3037] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:05.875553 12124 raft_consensus.cc:504] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.875998 12124 leader_election.cc:288] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [CANDIDATE]: Term 1 election: Requested vote from peers e1707ff37da44e1898373c325df053e7 (127.0.98.2:45887), 68780bb2d16d46cf9f8a7c7e6bf0d9d1 (127.0.98.3:39591)
I20260927 12:37:05.876768 11922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b727e7cd8441397aefa5be558b970" candidate_uuid: "186b54f598ba4ee59cb08e6df2065572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1707ff37da44e1898373c325df053e7"
I20260927 12:37:05.876909 11922 raft_consensus.cc:3037] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:05.878911 11922 raft_consensus.cc:2454] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b54f598ba4ee59cb08e6df2065572 in term 1.
I20260927 12:37:05.879419 11719 leader_election.cc:302] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186b54f598ba4ee59cb08e6df2065572, e1707ff37da44e1898373c325df053e7; no voters:
I20260927 12:37:05.879683 12124 raft_consensus.cc:2781] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:37:05.879913 12124 raft_consensus.cc:686] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [term 1 LEADER]: Becoming Leader. State: Replica: 186b54f598ba4ee59cb08e6df2065572, State: Running, Role: LEADER
I20260927 12:37:05.880422 12124 consensus_queue.cc:237] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.884424 11617 catalog_manager.cc:5495] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 reported cstate change: term changed from 0 to 1, leader changed from <none> to 186b54f598ba4ee59cb08e6df2065572 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "186b54f598ba4ee59cb08e6df2065572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } health_report { overall_health: UNKNOWN } } }
I20260927 12:37:05.887785 12128 raft_consensus.cc:482] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:37:05.887991 12128 raft_consensus.cc:504] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 } } peers { permanent_uuid: "186b54f598ba4ee59cb08e6df2065572" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34367 } } peers { permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 } }
I20260927 12:37:05.889321 12128 leader_election.cc:288] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1707ff37da44e1898373c325df053e7 (127.0.98.2:45887), 186b54f598ba4ee59cb08e6df2065572 (127.0.98.1:34367)
I20260927 12:37:05.894223 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.889691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45887 (local address 127.0.98.3:47251)
0927 12:37:05.889832 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.889848 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.889951 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.890397 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.890400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.890417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.890526 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.890532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.891562 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.891565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.892809 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.892820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.893781 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.893785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.893891 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.893914 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.893984 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.894064 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 12:37:05.895493 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.889742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47251 (local address 127.0.98.2:45887)
0927 12:37:05.889855 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.889860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.889905 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:37:05.890152 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.890156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.890222 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.890292 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.890645 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.891435 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.892949 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.893541 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.893644 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.893718 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.895262 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.895373 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:37:05.896240 11922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b727e7cd8441397aefa5be558b970" candidate_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1707ff37da44e1898373c325df053e7" is_pre_election: true
I20260927 12:37:05.896468 11922 raft_consensus.cc:2379] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68780bb2d16d46cf9f8a7c7e6bf0d9d1 in current term 1: Already voted for candidate 186b54f598ba4ee59cb08e6df2065572 in this term.
W20260927 12:37:05.897442 11969 tablet.cc:2369] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:37:05.901513 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.889958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34367 (local address 127.0.98.3:37451)
0927 12:37:05.894413 (+ 4455us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.894428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.894514 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.896021 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.896024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.896044 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.897425 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.897433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.898682 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.898686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.900053 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.900064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.901064 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.901067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.901148 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.901172 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.901252 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.901334 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4394,"thread_start_us":134,"threads_started":1}
I20260927 12:37:05.902057 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.890038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37451 (local address 127.0.98.1:34367)
0927 12:37:05.890155 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.890160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.894683 (+ 4523us) server_negotiation.cc:407] Connection header received
0927 12:37:05.894743 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.894746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.894800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.894872 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.897598 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.898423 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.900257 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.900826 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.900904 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.900974 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.901861 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.901946 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:37:05.902731 11771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b727e7cd8441397aefa5be558b970" candidate_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186b54f598ba4ee59cb08e6df2065572" is_pre_election: true
I20260927 12:37:05.903506 11992 leader_election.cc:302] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68780bb2d16d46cf9f8a7c7e6bf0d9d1; no voters: 186b54f598ba4ee59cb08e6df2065572, e1707ff37da44e1898373c325df053e7
I20260927 12:37:05.904148 12128 raft_consensus.cc:3037] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:05.904800 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.865779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38103 (local address 127.0.98.3:39591)
0927 12:37:05.865902 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.865908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.872538 (+ 6630us) server_negotiation.cc:407] Connection header received
0927 12:37:05.872731 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.872735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.872788 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.872833 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:05.888447 (+ 15614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.889202 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.903645 (+ 14443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.904187 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.904272 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.904328 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.904624 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.904698 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:37:05.905153 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.865655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39591 (local address 127.0.98.1:38103)
0927 12:37:05.872458 (+ 6803us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:05.872473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:05.872618 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:05.888053 (+ 15435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:05.888057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:05.888082 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:05.888299 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.888307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.889389 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.889393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.903517 (+ 14124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:05.903528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.904415 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.904418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:05.904499 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.904521 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:37:05.904954 (+ 433us) client_negotiation.cc:240] Negotiation successful
0927 12:37:05.905033 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6733,"thread_start_us":102,"threads_started":1}
I20260927 12:37:05.905527 12128 raft_consensus.cc:2726] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 12:37:05.905726 12052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b727e7cd8441397aefa5be558b970" candidate_uuid: "186b54f598ba4ee59cb08e6df2065572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" is_pre_election: true
I20260927 12:37:05.905954 12052 raft_consensus.cc:2454] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186b54f598ba4ee59cb08e6df2065572 in term 1.
I20260927 12:37:05.906261 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149b727e7cd8441397aefa5be558b970" candidate_uuid: "186b54f598ba4ee59cb08e6df2065572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1"
I20260927 12:37:05.907485 12042 raft_consensus.cc:2454] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186b54f598ba4ee59cb08e6df2065572 in term 1.
I20260927 12:37:05.954479 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:05.949762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.0.98.1:34367)
0927 12:37:05.949885 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:05.949890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:05.949906 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:37:05.949993 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:05.949996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:05.950064 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:05.950166 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:37:05.950580 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.951359 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.952591 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:05.952920 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:05.953025 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:05.953520 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:37:05.954026 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:37:05.954141 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:05.954274 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 12:37:05.954332 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:37:05.963056 11922 raft_consensus.cc:1261] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 1 FOLLOWER]: Refusing update from remote peer 186b54f598ba4ee59cb08e6df2065572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:37:05.963092 12042 raft_consensus.cc:1261] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 1 FOLLOWER]: Refusing update from remote peer 186b54f598ba4ee59cb08e6df2065572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:37:05.963753 12133 consensus_queue.cc:1035] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1707ff37da44e1898373c325df053e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:37:05.963982 12124 consensus_queue.cc:1035] T 149b727e7cd8441397aefa5be558b970 P 186b54f598ba4ee59cb08e6df2065572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68780bb2d16d46cf9f8a7c7e6bf0d9d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:37:05.981190 12139 mvcc.cc:205] Tried to move back new op lower bound from 7333939715937259520 to 7333939715610488832. Current Snapshot: MvccSnapshot[applied={T|T < 7333939715937259520}]
I20260927 12:37:05.984203 12137 mvcc.cc:205] Tried to move back new op lower bound from 7333939715937259520 to 7333939715610488832. Current Snapshot: MvccSnapshot[applied={T|T < 7333939715937259520}]
I20260927 12:37:05.985796 12138 mvcc.cc:205] Tried to move back new op lower bound from 7333939715937259520 to 7333939715610488832. Current Snapshot: MvccSnapshot[applied={T|T < 7333939715937259520}]
I20260927 12:37:05.996327 11617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:37:05.996523 11617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:37:05.999456 11617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6d20ee0b0a2499ca380693ecb1aeb23: Sending DeleteTablet for 3 replicas of tablet 149b727e7cd8441397aefa5be558b970
I20260927 12:37:06.000916 11902 tablet_service.cc:1514] Processing DeleteTablet for tablet 149b727e7cd8441397aefa5be558b970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:37:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51852
I20260927 12:37:06.001564 11763 tablet_service.cc:1514] Processing DeleteTablet for tablet 149b727e7cd8441397aefa5be558b970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:37:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40480
I20260927 12:37:06.002524 12036 tablet_service.cc:1514] Processing DeleteTablet for tablet 149b727e7cd8441397aefa5be558b970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:37:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57526
I20260927 12:37:06.003839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 11699
I20260927 12:37:06.010198 12148 tablet_replica.cc:330] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: stopping tablet replica
I20260927 12:37:06.010727 12146 tablet_replica.cc:330] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7: stopping tablet replica
I20260927 12:37:06.011142 12148 raft_consensus.cc:2229] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:37:06.011142 12146 raft_consensus.cc:2229] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:37:06.011634 12146 raft_consensus.cc:2258] T 149b727e7cd8441397aefa5be558b970 P e1707ff37da44e1898373c325df053e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:37:06.014704 12148 raft_consensus.cc:2258] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:37:06.018301 12148 ts_tablet_manager.cc:1905] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:37:06.018446 11605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34367 (error 108)
I20260927 12:37:06.019398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 11838
I20260927 12:37:06.022010 12148 ts_tablet_manager.cc:1918] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:37:06.022054 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:06.021649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34367 (local address 127.0.0.1:40500)
0927 12:37:06.021869 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:06.021951 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 12:37:06.022135 12148 log.cc:1196] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/149b727e7cd8441397aefa5be558b970
I20260927 12:37:06.022593 12148 ts_tablet_manager.cc:1939] T 149b727e7cd8441397aefa5be558b970 P 68780bb2d16d46cf9f8a7c7e6bf0d9d1: Deleting consensus metadata
W20260927 12:37:06.022542 11605 catalog_manager.cc:4584] TS 186b54f598ba4ee59cb08e6df2065572 (127.0.98.1:34367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 149b727e7cd8441397aefa5be558b970: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34367: connect: Connection refused (error 111)
I20260927 12:37:06.025616 11603 catalog_manager.cc:4841] TS 68780bb2d16d46cf9f8a7c7e6bf0d9d1 (127.0.98.3:39591): tablet 149b727e7cd8441397aefa5be558b970 (table test-table [id=6bed3715e2bd4fcd97d8ed15b27ed2b7]) successfully deleted
I20260927 12:37:06.026240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 11973
I20260927 12:37:06.026996 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:06.026457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45887 (local address 127.0.0.1:51854)
0927 12:37:06.026847 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:06.026907 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290}
I20260927 12:37:06.033459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 11586
I20260927 12:37:06.039559 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 12:37:06.375677 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 12:37:06.375829 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 12:37:06.711467 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 12:37:06.711 UTC [11172] LOG: received smart shutdown request
2026-09-27 12:37:06.716 UTC [11172] LOG: background worker "logical replication launcher" (PID 11181) exited with exit code 1
2026-09-27 12:37:06.716 UTC [11176] LOG: shutting down
2026-09-27 12:37:06.952 UTC [11172] LOG: database system is shut down
2026-09-27T12:37:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50022 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:37:07 dist-test-slave-tzgx krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:37:07 dist-test-slave-tzgx krb5kdc[12158](info): set up 2 sockets
Sep 27 12:37:07 dist-test-slave-tzgx krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260927 12:37:08.040045 392 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:37:08 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:37:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 12:37:08.224223 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRFp9Fb/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/postgres -l logfile start
2026-09-27 12:37:11.296 UTC [12188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 12:37:11.296 UTC [12188] LOG: listening on IPv4 address "127.0.98.20", port 42913
2026-09-27 12:37:11.301 UTC [12188] LOG: listening on Unix socket "/tmp/.s.PGSQL.42913"
2026-09-27 12:37:11.322 UTC [12191] LOG: database system was shut down at 2026-09-27 12:37:09 UTC
2026-09-27 12:37:11.326 UTC [12188] LOG: database system is ready to accept connections
I20260927 12:37:12.211740 392 mini_postgres.cc:96] Postgres bound to 42913
2026-09-27 12:37:12.218 UTC [12199] FATAL: database "slave" does not exist
127.0.98.20:42913 - accepting connections
I20260927 12:37:12.218874 392 mini_ranger.cc:161] Starting Ranger...
I20260927 12:37:12.234228 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 12:37:12.864014 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 12:37:12.864117 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-admin
2026-09-27 12:37:13,124 [I] DB FLAVOR :POSTGRES
2026-09-27 12:37:13,125 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:37:13,125 [I] Checking connection..
2026-09-27 12:37:13,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 12:37:13,396 [I] Checking connection passed.
2026-09-27 12:37:13,397 [I] --------- Verifying version history table ---------
2026-09-27 12:37:13,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:37:13,664 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:37:13,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:37:13,915 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:37:13,915 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 12:37:13,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 12:37:14.186 UTC [12263] WARNING: there is no transaction in progress
2026-09-27 12:37:14,192 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 12:37:14,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:37:14,443 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 12:37:14,443 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:37:14,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 12:37:14,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 12:37:14,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tzgx.c.gcp-upstream.internal','N') ;"
2026-09-27 12:37:15,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:37:15,428 [I] Table x_portal_user does not exist in database ranger
2026-09-27 12:37:15,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:37:15,687 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 12:37:15,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:37:15,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:37:16,220 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 12:37:16,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 12:37:17.461 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.481 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.502 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.523 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.543 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.569 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.638 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.646 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.664 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.676 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.689 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.705 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.716 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.725 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.733 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:17.743 UTC [12371] WARNING: there is no transaction in progress
2026-09-27 12:37:18,591 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 12:37:18,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 12:37:18,843 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 12:37:18,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:37:19,102 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 12:37:19,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:37:19,351 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 12:37:19,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:37:19,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:37:19,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tzgx.c.gcp-upstream.internal';"
2026-09-27 12:37:20,159 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 12:37:20,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 12:37:20,429 [I] DB_PATCHES have already been applied
I20260927 12:37:20.435462 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRFp9Fb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 12:37:20.435536 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 12:37:20.439620 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38119
Sep 27, 2026 12:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:37:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 12:37:21.450440 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.232s user 0.000s sys 0.011s
Sep 27, 2026 12:37:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:37:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 12:37:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:37:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:37:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 12:37:43.646046 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 12:37:43 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 12:37:43.739727 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 12:37:43.749014 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 12:37:44.806088 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 12:37:44.806161 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms
I20260927 12:37:44.806180 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42913
2026-09-27 12:37:44,967 [I] DB FLAVOR :POSTGRES
2026-09-27 12:37:44,967 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:37:44,967 [I] Checking connection
2026-09-27 12:37:44,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 12:37:45,242 [I] connection success
2026-09-27 12:37:45,242 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 12:37:45,243 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 12:37:45,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 12:37:45,521 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 12:37:45,521 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:37:45,521 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 12:37:45,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42913/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 12:37:45,807 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 12:37:46.326390 392 mini_ranger_kms.cc:310] Created kms service
I20260927 12:37:46.583343 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 12:37:46.631781 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 12:37:46 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:47.095378 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 12:37:47.095539 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRFp9Fb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRFp9Fb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 12:37:47.095558 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 12:37:47.099870 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12552
I20260927 12:37:47.099953 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55533
Sep 27, 2026 12:37:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:37:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:37:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:37:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:37:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:37:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 12:37:49.222534 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.483s user 0.015s sys 0.130s
I20260927 12:37:49.222638 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 12:37:49.222711 392 mini_ranger_kms.cc:401] 127.0.98.20:55533/kms/v1/keys
Sep 27 12:37:49 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512669, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 12:37:50 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512669, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:37:50 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512669, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:37:52 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:37:52.720283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34107
--ranger_config_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:52.862381 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:52.862691 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:52.862756 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:52.867053 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:37:52.867138 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:52.867165 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:52.867185 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:37:52.867216 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:37:52.873588 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55533/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34107
--ranger_config_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:52.874754 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:52.876024 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:52.888227 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:52.888233 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:52.888427 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:52.888478 12602 server_base.cc:1034] running on GCE node
I20260927 12:37:52.889209 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:52.890487 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:52.891693 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790512672891678 us; error 36 us; skew 500 ppm
Sep 27 12:37:52 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:52.901527 12602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:37:52.902743 12602 webserver.cc:460] Webserver started at http://127.0.98.62:46029/ using document root <none> and password file <none>
I20260927 12:37:52.903342 12602 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:52.903404 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:52.903820 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:52 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:52.968999 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "b4bf5d2f0f5e468f82414202a7afa3b8"
format_stamp: "Formatted at 2026-09-27 12:37:52 on dist-test-slave-tzgx"
server_key: "04e3a8ee9801bfa32b7111cc61bbd5f3"
server_key_iv: "eabd7381bdfcfe930b05ef0ef609e4fa"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:52.969666 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4bf5d2f0f5e468f82414202a7afa3b8"
format_stamp: "Formatted at 2026-09-27 12:37:52 on dist-test-slave-tzgx"
server_key: "04e3a8ee9801bfa32b7111cc61bbd5f3"
server_key_iv: "eabd7381bdfcfe930b05ef0ef609e4fa"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:52.973933 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:37:52.992400 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:52.993748 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:37:52.993881 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "b4bf5d2f0f5e468f82414202a7afa3b8"
format_stamp: "Formatted at 2026-09-27 12:37:52 on dist-test-slave-tzgx"
server_key: "04e3a8ee9801bfa32b7111cc61bbd5f3"
server_key_iv: "eabd7381bdfcfe930b05ef0ef609e4fa"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:52.994015 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:53.030102 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:53.033182 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:53.033360 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:53.041116 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34107
I20260927 12:37:53.041121 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34107 every 8 connection(s)
I20260927 12:37:53.042177 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:37:53.045080 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:53.050081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 12602
I20260927 12:37:53.050212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:37:53.050513 392 external_mini_cluster.cc:1428] Setting key 2ec982c4b22b9589015b3be64b91ffd9
I20260927 12:37:53.051116 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Bootstrap starting.
I20260927 12:37:53.054049 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:53.054981 12670 log.cc:824] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:53.057524 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: No bootstrap required, opened a new log
Sep 27 12:37:53 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:53.062474 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:53.062811 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:53.062901 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bf5d2f0f5e468f82414202a7afa3b8, State: Initialized, Role: FOLLOWER
I20260927 12:37:53.063462 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:53.063654 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:37:53.063746 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:37:53.063902 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:53.065106 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:53.065486 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:53.052211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.0.98.62:34107)
0927 12:37:53.052740 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:53.052760 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:53.052803 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:37:53.053681 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:53.053706 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:53.054153 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:53.054543 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:53.055448 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:53.056619 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:53.057524 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:53.058072 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:53.058233 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:53.060712 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:53.060719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:53.060735 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:53.060767 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:53.062302 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:53.062745 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:53.062750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:53.062755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:53.062893 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:53.063203 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:53.063206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:53.063208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:53.063707 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:53.063908 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:53.064437 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 12:37:53.064696 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":168,"threads_started":1}
I20260927 12:37:53.065513 12670 leader_election.cc:302] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4bf5d2f0f5e468f82414202a7afa3b8; no voters:
I20260927 12:37:53.065879 12670 leader_election.cc:288] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:37:53.066278 12676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:37:53.066515 12676 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 LEADER]: Becoming Leader. State: Replica: b4bf5d2f0f5e468f82414202a7afa3b8, State: Running, Role: LEADER
I20260927 12:37:53.066879 12676 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:53.067252 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:37:53.068185 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } } }
I20260927 12:37:53.068342 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: This master's current role is: LEADER
I20260927 12:37:53.068558 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bf5d2f0f5e468f82414202a7afa3b8. Latest consensus state: current_term: 1 leader_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } } }
I20260927 12:37:53.068670 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: This master's current role is: LEADER
I20260927 12:37:53.069695 12670 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:37:53.072566 12684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:37:53.077561 12684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:37:53.086150 12684 catalog_manager.cc:1347] Generated new cluster ID: 7c29473c4b8a46238856e54d31fac280
I20260927 12:37:53.086228 12684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:37:53.103322 12684 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:37:53.104828 12684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:37:53.115342 12684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Generated new TSK 0
I20260927 12:37:53.116468 12684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:37:53.844029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34107
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:53.977080 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:53.977376 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:53.977435 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:53.981871 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:53.981938 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:53.982019 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:37:53.987613 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:53.988734 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:53.989862 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:54.000979 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:54.001044 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:54.000993 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:54.001441 12715 server_base.cc:1034] running on GCE node
I20260927 12:37:54.001981 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:54.002899 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:54.004158 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790512674004141 us; error 35 us; skew 500 ppm
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:54.007650 12715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:37:54.008787 12715 webserver.cc:460] Webserver started at http://127.0.98.1:43429/ using document root <none> and password file <none>
I20260927 12:37:54.009529 12715 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:54.009658 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:54.009964 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:54.030124 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "dc4cab897f2b4563b510e9740ff4dc19"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "98bb13301815fd2d4acbdee94415f6f3"
server_key_iv: "71968238554e0417f872675268e5a979"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.030946 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dc4cab897f2b4563b510e9740ff4dc19"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "98bb13301815fd2d4acbdee94415f6f3"
server_key_iv: "71968238554e0417f872675268e5a979"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.036412 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 12:37:54.067799 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.069947 12715 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260927 12:37:54.070228 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "dc4cab897f2b4563b510e9740ff4dc19"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "98bb13301815fd2d4acbdee94415f6f3"
server_key_iv: "71968238554e0417f872675268e5a979"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.070405 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:54.120296 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:54.123409 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:54.123602 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:54.124207 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:54.125083 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:54.125190 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.125321 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:54.125352 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.137553 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44461
I20260927 12:37:54.139418 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:37:54.142246 12847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44461 every 8 connection(s)
I20260927 12:37:54.146546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 12715
I20260927 12:37:54.146661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:37:54.146922 392 external_mini_cluster.cc:1428] Setting key b291391a323fd70760e1f4c36e3fdcd9
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:54.167403 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:54.144588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34107 (local address 127.0.98.1:46229)
0927 12:37:54.150999 (+ 6411us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:54.151085 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:54.152573 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:54.153535 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:54.153551 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:54.153948 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:54.154670 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:54.154692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.155933 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.155936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:54.157411 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:54.157423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.158251 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.158254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:54.158465 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:54.159140 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:54.159163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:54.161249 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:54.163770 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:54.163778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:54.163798 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:54.164232 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:54.164561 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:54.164564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:54.164566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:54.164762 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:54.165277 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:54.165292 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:54.165677 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 12:37:54.165886 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 12:37:54.166267 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6090,"thread_start_us":150,"threads_started":1}
I20260927 12:37:54.168318 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:54.145793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46229 (local address 127.0.98.62:34107)
0927 12:37:54.146043 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:54.146048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:54.151293 (+ 5245us) server_negotiation.cc:407] Connection header received
0927 12:37:54.152888 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:54.152892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:54.152953 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:54.153018 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:54.154870 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.155761 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.157557 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.157927 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.158017 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:54.161496 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:54.161501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:54.161504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:54.161537 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:54.163605 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:54.164360 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:54.164364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:54.164366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:54.164464 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:54.164869 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:54.164872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:54.164874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:54.165105 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:54.165200 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:54.167947 (+ 2747us) server_negotiation.cc:299] Negotiation successful
0927 12:37:54.168144 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
I20260927 12:37:54.170217 12848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34107
I20260927 12:37:54.170476 12848 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:54.171054 12848 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:54.173182 12627 ts_manager.cc:194] Registered new tserver with Master: dc4cab897f2b4563b510e9740ff4dc19 (127.0.98.1:44461)
I20260927 12:37:54.174935 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:37:54.180285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34107
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 12:37:54.342087 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:54.342520 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:54.342602 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:54.348950 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:54.349038 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:54.349139 12854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:37:54.357498 12854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:54.359061 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:54.360519 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:54.373356 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:54.374281 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:54.374864 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:54.375391 12854 server_base.cc:1034] running on GCE node
I20260927 12:37:54.376125 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:54.377126 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:54.378504 12854 hybrid_clock.cc:648] HybridClock initialized: now 1790512674378476 us; error 47 us; skew 500 ppm
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:54.382471 12854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:37:54.383927 12854 webserver.cc:460] Webserver started at http://127.0.98.2:45763/ using document root <none> and password file <none>
I20260927 12:37:54.384825 12854 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:54.384991 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:54.385313 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:54.409938 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "032c68861e1d411bbbd397c51917f0a9"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "5ca025a1eecf1b8cf6418886c7450e2e"
server_key_iv: "dcac1929c76f204746d3e06b9b4a264d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.411940 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "032c68861e1d411bbbd397c51917f0a9"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "5ca025a1eecf1b8cf6418886c7450e2e"
server_key_iv: "dcac1929c76f204746d3e06b9b4a264d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.417025 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 12:37:54.448123 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.453816 12854 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.002s sys 0.000s
I20260927 12:37:54.454007 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "032c68861e1d411bbbd397c51917f0a9"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "5ca025a1eecf1b8cf6418886c7450e2e"
server_key_iv: "dcac1929c76f204746d3e06b9b4a264d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.454177 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:54.483963 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:54.487550 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:54.487767 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:54.488543 12854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:54.489497 12854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:54.489547 12854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.489604 12854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:54.489634 12854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.499689 12854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38037
I20260927 12:37:54.499733 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38037 every 8 connection(s)
I20260927 12:37:54.500941 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:37:54.503657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 12854
I20260927 12:37:54.503784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:37:54.504154 392 external_mini_cluster.cc:1428] Setting key 768a0f8bc4e531a6dc6ba2aced6f2404
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:54.518836 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:54.503431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58615 (local address 127.0.98.62:34107)
0927 12:37:54.503584 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:54.503587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:54.504887 (+ 1300us) server_negotiation.cc:407] Connection header received
0927 12:37:54.506234 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:54.506238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:54.506292 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:54.506380 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:54.507657 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.508573 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.510528 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.510887 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.510982 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:54.514223 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:54.514227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:54.514230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:54.514264 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:54.516182 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:54.516884 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:54.516887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:54.516889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:54.516978 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:54.517396 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:54.517399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:54.517401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:54.517630 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:54.517699 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:54.518503 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 12:37:54.518661 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:37:54.520604 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:54.503975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34107 (local address 127.0.98.2:58615)
0927 12:37:54.504656 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:54.504729 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:54.505974 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:54.506585 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:54.506595 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:54.506897 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:54.507468 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:54.507484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.508760 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.508765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:54.510354 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:54.510370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.511187 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.511192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:54.511417 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:54.512162 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:54.512187 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:54.513957 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:54.516344 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:54.516352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:54.516375 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:54.516763 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:54.517100 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:54.517102 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:54.517105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:54.517301 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:54.517781 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:54.517804 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:54.518214 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 12:37:54.519276 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0927 12:37:54.519603 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":178,"threads_started":1}
I20260927 12:37:54.522284 12984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34107
I20260927 12:37:54.522500 12984 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:54.523175 12984 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:54.524737 12627 ts_manager.cc:194] Registered new tserver with Master: 032c68861e1d411bbbd397c51917f0a9 (127.0.98.2:38037)
I20260927 12:37:54.525619 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58615
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:37:54.547122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34107
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 12:37:54.691795 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:54.692618 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:54.692734 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:54.698897 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:54.699101 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:54.699244 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:37:54.707733 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:54.709481 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:54.711010 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:54.724622 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:54.725629 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:54.727131 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:54.729156 12989 server_base.cc:1034] running on GCE node
I20260927 12:37:54.729900 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:54.730831 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:54.734146 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790512674734048 us; error 101 us; skew 500 ppm
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:54.738386 12989 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:37:54.739710 12989 webserver.cc:460] Webserver started at http://127.0.98.3:34133/ using document root <none> and password file <none>
I20260927 12:37:54.740569 12989 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:54.740741 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:54.741042 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:54.757974 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9a027f4463f42abb10a848f067d56f0"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "4a996f7b42a9b6c159cf006493616d7f"
server_key_iv: "90282d98783dbf47c5e43dfe5cfefe88"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.758733 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9a027f4463f42abb10a848f067d56f0"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "4a996f7b42a9b6c159cf006493616d7f"
server_key_iv: "90282d98783dbf47c5e43dfe5cfefe88"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.768258 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260927 12:37:54.793677 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.795413 12989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:37:54.795563 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "c9a027f4463f42abb10a848f067d56f0"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "4a996f7b42a9b6c159cf006493616d7f"
server_key_iv: "90282d98783dbf47c5e43dfe5cfefe88"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:54.795722 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:54.814899 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:54.818703 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:54.818892 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:54.819577 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:54.820659 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:54.820721 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.820775 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:54.820796 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:54.831782 12989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40557
I20260927 12:37:54.831835 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40557 every 8 connection(s)
I20260927 12:37:54.833714 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:37:54.837795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 12989
I20260927 12:37:54.837919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:37:54.838229 392 external_mini_cluster.cc:1428] Setting key 60b3455168839ceb73e52a4eb94b4755
Sep 27 12:37:54 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:54.849756 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:54.836371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39665 (local address 127.0.98.62:34107)
0927 12:37:54.836513 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:54.836518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:54.837174 (+ 656us) server_negotiation.cc:407] Connection header received
0927 12:37:54.838446 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:54.838450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:54.838509 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:54.838560 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:54.840197 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.841030 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.842751 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.843107 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.843219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:54.846283 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:54.846287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:54.846289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:54.846316 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:54.847629 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:54.848194 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:54.848197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:54.848199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:54.848276 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:54.848615 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:54.848617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:54.848619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:54.848809 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:54.848892 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:54.849445 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 12:37:54.849571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:37:54.850764 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:54.836414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34107 (local address 127.0.98.3:39665)
0927 12:37:54.836990 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:54.837049 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:54.838187 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:54.838966 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:54.838974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:54.839334 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:54.839999 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:54.840018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.841182 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.841186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:54.842601 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:54.842613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:54.843283 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:54.843288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:54.843501 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:54.844273 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:54.844296 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:54.846069 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:54.847760 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:54.847768 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:54.847788 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:54.848063 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:54.848361 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:54.848365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:54.848367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:54.848520 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:54.848904 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:54.848912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:54.849204 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 12:37:54.849372 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 12:37:54.849709 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":156,"threads_started":1}
I20260927 12:37:54.852646 13118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34107
I20260927 12:37:54.852923 13118 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:54.853533 13118 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:54.854532 12634 ts_manager.cc:194] Registered new tserver with Master: c9a027f4463f42abb10a848f067d56f0 (127.0.98.3:40557)
I20260927 12:37:54.855149 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39665
I20260927 12:37:54.865350 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:37:54.865931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 12715
I20260927 12:37:54.872426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 12854
I20260927 12:37:54.877630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 12989
I20260927 12:37:54.882555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 12602
I20260927 12:37:54.888363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34107
--webserver_interface=127.0.98.62
--webserver_port=46029
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34107
--ranger_config_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:55.000519 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:55.000799 13121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:55.000839 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:55.004958 13121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:37:55.005028 13121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:55.005048 13121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:55.005062 13121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:37:55.005076 13121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:37:55.010771 13121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34107
--ranger_config_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:55.011802 13121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:55.012780 13121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:55.021673 13129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:55.021716 13127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:55.021689 13126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.021698 13121 server_base.cc:1034] running on GCE node
I20260927 12:37:55.022692 13121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:55.023921 13121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:55.025110 13121 hybrid_clock.cc:648] HybridClock initialized: now 1790512675025080 us; error 49 us; skew 500 ppm
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:55.027917 13121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:37:55.028949 13121 webserver.cc:460] Webserver started at http://127.0.98.62:46029/ using document root <none> and password file <none>
I20260927 12:37:55.029472 13121 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:55.029539 13121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:55.076615 13121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:37:55.078895 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.080417 13121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:37:55.080547 13121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "b4bf5d2f0f5e468f82414202a7afa3b8"
format_stamp: "Formatted at 2026-09-27 12:37:52 on dist-test-slave-tzgx"
server_key: "04e3a8ee9801bfa32b7111cc61bbd5f3"
server_key_iv: "eabd7381bdfcfe930b05ef0ef609e4fa"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:55.081139 13121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:55.103915 13121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:55.106923 13121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:55.107086 13121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:55.114276 13121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34107
I20260927 12:37:55.114367 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34107 every 8 connection(s)
I20260927 12:37:55.115407 13121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:37:55.115762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13121
I20260927 12:37:55.116236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44461
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34107
--webserver_port=43429
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:37:55.120242 13190 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:37:55.122192 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Bootstrap starting.
I20260927 12:37:55.139242 13190 log.cc:824] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:55.146235 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:37:55.146634 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Bootstrap complete.
I20260927 12:37:55.150681 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:55.151136 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bf5d2f0f5e468f82414202a7afa3b8, State: Initialized, Role: FOLLOWER
I20260927 12:37:55.151821 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:55.152009 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:37:55.152139 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:37:55.152257 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:37:55.154311 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:55.154666 13190 leader_election.cc:302] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4bf5d2f0f5e468f82414202a7afa3b8; no voters:
I20260927 12:37:55.154986 13190 leader_election.cc:288] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:37:55.155184 13194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:37:55.155511 13194 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [term 2 LEADER]: Becoming Leader. State: Replica: b4bf5d2f0f5e468f82414202a7afa3b8, State: Running, Role: LEADER
I20260927 12:37:55.155859 13194 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } }
I20260927 12:37:55.156518 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:37:55.157281 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } } }
I20260927 12:37:55.157410 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: This master's current role is: LEADER
I20260927 12:37:55.158188 13201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:37:55.158342 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bf5d2f0f5e468f82414202a7afa3b8. Latest consensus state: current_term: 2 leader_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bf5d2f0f5e468f82414202a7afa3b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34107 } } }
I20260927 12:37:55.158504 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8 [sys.catalog]: This master's current role is: LEADER
I20260927 12:37:55.161523 13201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:37:55.162200 13201 catalog_manager.cc:1259] Loaded cluster ID: 7c29473c4b8a46238856e54d31fac280
I20260927 12:37:55.162290 13201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:37:55.162196 13190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:37:55.167933 13201 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:37:55.169778 13201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Loaded TSK: 0
I20260927 12:37:55.171551 13201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:37:55.261018 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:55.261380 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:55.261449 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:55.266139 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:55.266217 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:55.266304 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:37:55.271946 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43429
--tserver_master_addrs=127.0.98.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:55.273088 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:55.274194 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:55.287317 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.287797 13192 server_base.cc:1034] running on GCE node
W20260927 12:37:55.287936 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:55.288276 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.289017 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:55.289880 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:55.291064 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790512675291034 us; error 45 us; skew 500 ppm
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:55.296355 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:37:55.297761 13192 webserver.cc:460] Webserver started at http://127.0.98.1:43429/ using document root <none> and password file <none>
I20260927 12:37:55.298621 13192 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:55.298776 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:55.347733 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 12:37:55.352989 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.354725 13192 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260927 12:37:55.354856 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "dc4cab897f2b4563b510e9740ff4dc19"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "98bb13301815fd2d4acbdee94415f6f3"
server_key_iv: "71968238554e0417f872675268e5a979"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:55.355492 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:55.397783 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:55.403352 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:55.403581 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:55.404366 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:55.405498 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:55.405571 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.405627 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:55.405663 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.416998 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44461
I20260927 12:37:55.417092 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44461 every 8 connection(s)
I20260927 12:37:55.418293 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:37:55.420560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13192
I20260927 12:37:55.421026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38037
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34107
--webserver_port=45763
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:55.449013 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.422556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33221 (local address 127.0.98.62:34107)
0927 12:37:55.424447 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:55.424472 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:55.424524 (+ 52us) server_negotiation.cc:407] Connection header received
0927 12:37:55.424873 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:55.424893 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:55.425413 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:55.425861 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:55.427174 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.428870 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.430669 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.431453 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.431639 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.435687 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:55.435695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:55.435718 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:55.435757 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:55.439339 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.440177 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.440184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.440190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.440353 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.440802 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.440806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.440808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.441358 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:55.441484 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:55.447736 (+ 6252us) server_negotiation.cc:299] Negotiation successful
0927 12:37:55.448017 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688,"thread_start_us":114,"threads_started":1}
I20260927 12:37:55.449028 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.421082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34107 (local address 127.0.98.1:33221)
0927 12:37:55.421599 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:55.421654 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:55.422681 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:55.426127 (+ 3446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:55.426135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:55.426428 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:55.426963 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:55.426978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.429081 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.429085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:55.430501 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:55.430515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.431907 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.431911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:55.432157 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.432891 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:55.432913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:55.435371 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:55.439520 (+ 4149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:55.439528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:55.439546 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:55.439974 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:55.440479 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:55.440483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:55.440485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:55.440681 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:55.441557 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:55.441564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:55.447246 (+ 5682us) client_negotiation.cc:769] Sending connection context
0927 12:37:55.447542 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 12:37:55.447952 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260927 12:37:55.452176 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34107
I20260927 12:37:55.452455 13352 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.453205 13352 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
W20260927 12:37:55.456043 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:37:55.456231 13352 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.456440 13352 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:55.457676 13352 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.457903 13352 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
W20260927 12:37:55.458866 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:37:55.568750 13355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:55.569033 13355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:55.569096 13355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:55.574540 13355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:55.574647 13355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:55.574744 13355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:37:55.580850 13355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45763
--tserver_master_addrs=127.0.98.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:55.582003 13355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:55.583428 13355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:55.596391 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:55.597014 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.597292 13355 server_base.cc:1034] running on GCE node
W20260927 12:37:55.597292 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.597949 13355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:55.598776 13355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:55.602077 13355 hybrid_clock.cc:648] HybridClock initialized: now 1790512675601985 us; error 102 us; skew 500 ppm
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:55.606034 13355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:37:55.607306 13355 webserver.cc:460] Webserver started at http://127.0.98.2:45763/ using document root <none> and password file <none>
I20260927 12:37:55.608129 13355 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:55.608248 13355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:55.629819 13355 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:37:55.632754 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.637629 13355 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260927 12:37:55.637802 13355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "032c68861e1d411bbbd397c51917f0a9"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "5ca025a1eecf1b8cf6418886c7450e2e"
server_key_iv: "dcac1929c76f204746d3e06b9b4a264d"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:55.638615 13355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:55.655063 13355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:55.658383 13355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:55.658603 13355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:55.659319 13355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:55.660312 13355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:55.660363 13355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.660409 13355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:55.660439 13355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.670524 13355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38037
I20260927 12:37:55.670849 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38037 every 8 connection(s)
I20260927 12:37:55.671854 13355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:37:55.681802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13355
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:55.682219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40557
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34107
--webserver_port=34133
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:37:55.688644 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.673954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58211 (local address 127.0.98.62:34107)
0927 12:37:55.674364 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:55.674369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:55.675029 (+ 660us) server_negotiation.cc:407] Connection header received
0927 12:37:55.676052 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:55.676056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:55.676146 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:55.676221 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:55.677320 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.677854 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.678986 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.679226 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.679314 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.683419 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:55.683424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:55.683426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:55.683465 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:55.685473 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.686211 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.686215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.686217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.686326 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.686815 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.686819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.686821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.687306 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:55.687394 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:55.688288 (+ 894us) server_negotiation.cc:299] Negotiation successful
0927 12:37:55.688450 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260927 12:37:55.690374 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.674406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34107 (local address 127.0.98.2:58211)
0927 12:37:55.674881 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:55.674932 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:55.675839 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:55.676413 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:55.676420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:55.676695 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:55.677172 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:55.677186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.677977 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.677979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:55.678864 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:55.678874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.679372 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.679376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:55.679556 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.680925 (+ 1369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:55.680945 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:55.683169 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:55.685611 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:55.685619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:55.685639 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:55.686061 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:55.686464 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:55.686468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:55.686470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:55.686685 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:55.687488 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:55.687496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:55.687889 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 12:37:55.688853 (+ 964us) client_negotiation.cc:240] Negotiation successful
0927 12:37:55.689256 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":105,"threads_started":1}
I20260927 12:37:55.691833 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34107
I20260927 12:37:55.692166 13486 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.692863 13486 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
W20260927 12:37:55.694852 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:37:55.694991 13486 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.695202 13486 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:55.696213 13486 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.696411 13486 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
W20260927 12:37:55.697281 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:37:55.833101 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:55.833403 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:55.833464 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:55.837754 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:55.837821 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:55.837905 13490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:37:55.843433 13490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34133
--tserver_master_addrs=127.0.98.62:34107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:55.844717 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:55.846131 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:55.859141 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.859297 13490 server_base.cc:1034] running on GCE node
W20260927 12:37:55.859148 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:55.859522 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:55.860023 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:55.860971 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:55.862174 13490 hybrid_clock.cc:648] HybridClock initialized: now 1790512675862151 us; error 66 us; skew 500 ppm
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:55.866250 13490 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:37:55.867580 13490 webserver.cc:460] Webserver started at http://127.0.98.3:34133/ using document root <none> and password file <none>
I20260927 12:37:55.868418 13490 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:55.868546 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:55.893085 13490 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:37:55.897233 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.898674 13490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:37:55.898803 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "c9a027f4463f42abb10a848f067d56f0"
format_stamp: "Formatted at 2026-09-27 12:37:54 on dist-test-slave-tzgx"
server_key: "4a996f7b42a9b6c159cf006493616d7f"
server_key_iv: "90282d98783dbf47c5e43dfe5cfefe88"
server_key_version: "kuduclusterkey@0"
I20260927 12:37:55.899387 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:55.914484 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:55.917802 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:55.918004 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:55.918696 13490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:37:55.919715 13490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:37:55.919772 13490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.919831 13490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:37:55.919860 13490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:55.930513 13490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40557
I20260927 12:37:55.932009 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:37:55.938547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13490
I20260927 12:37:55.940253 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40557 every 8 connection(s)
Sep 27 12:37:55 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:55.955808 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.939311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.0.98.62:34107)
0927 12:37:55.939624 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:55.939629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:55.944540 (+ 4911us) server_negotiation.cc:407] Connection header received
0927 12:37:55.944713 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:55.944716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:55.944785 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:55.944958 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:55.947380 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.948256 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.949459 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.949809 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.949910 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.951832 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:55.951836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:55.951838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:55.951866 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:55.953707 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.954222 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.954226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.954228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.954323 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.954653 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.954657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.954659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.954877 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:55.955083 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:55.955317 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 12:37:55.955453 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260927 12:37:55.959946 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.936663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33417 (local address 127.0.98.62:34107)
0927 12:37:55.936797 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:55.936802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:55.947825 (+ 11023us) server_negotiation.cc:407] Connection header received
0927 12:37:55.949037 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:55.949041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:55.949100 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:55.949155 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:55.950791 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.951563 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.953100 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.953334 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.953428 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.956076 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:55.956080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:55.956083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:55.956131 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:55.957828 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.958361 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.958365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.958367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.958449 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:55.958803 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:55.958807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:55.958809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:55.959006 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:55.959117 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:55.959653 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 12:37:55.959778 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:37:55.960839 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:55.936251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34107 (local address 127.0.98.3:33417)
0927 12:37:55.947646 (+ 11395us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:55.947709 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:55.948745 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:55.949603 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:55.949611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:55.949943 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:55.950611 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:55.950633 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.951748 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.951752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:55.952968 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:55.952980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:55.953488 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:55.953494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:55.953691 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:55.954263 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:55.954281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:55.955856 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:55.957939 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:55.957946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:55.957964 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:55.958254 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:55.958531 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:55.958534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:55.958537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:55.958710 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:55.959115 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:55.959123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:55.959410 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 12:37:55.959599 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:37:55.959927 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11133,"thread_start_us":1140,"threads_started":1}
I20260927 12:37:55.962134 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34107
I20260927 12:37:55.962399 13620 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.962888 13620 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
W20260927 12:37:55.964414 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:37:55.964562 13620 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.964752 13620 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:55.965699 13620 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:55.965891 13620 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
W20260927 12:37:55.966728 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 12:37:56 dist-test-slave-tzgx krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:56.459143 13352 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:56.459422 13352 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:56.460788 13154 ts_manager.cc:194] Registered new tserver with Master: dc4cab897f2b4563b510e9740ff4dc19 (127.0.98.1:44461)
I20260927 12:37:56.462255 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33221
I20260927 12:37:56.697539 13486 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:56.697860 13486 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:56.698820 13154 ts_manager.cc:194] Registered new tserver with Master: 032c68861e1d411bbbd397c51917f0a9 (127.0.98.2:38037)
I20260927 12:37:56.699676 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58211
Sep 27 12:37:56 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:56.966954 13620 heartbeater.cc:464] Registering TS with master...
I20260927 12:37:56.967239 13620 heartbeater.cc:511] Master 127.0.98.62:34107 requested a full tablet report, sending...
I20260927 12:37:56.967945 13154 ts_manager.cc:194] Registered new tserver with Master: c9a027f4463f42abb10a848f067d56f0 (127.0.98.3:40557)
I20260927 12:37:56.968765 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33417
I20260927 12:37:56.978611 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:37:56 dist-test-slave-tzgx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:37:57.003046 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:56.982050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.62:34107)
0927 12:37:56.992335 (+ 10285us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:56.992341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:56.992361 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:37:56.992418 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:56.992422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:56.992485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:56.992590 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:56.993616 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:56.994442 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:56.995967 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:56.996353 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:56.996461 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:56.998154 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:56.998159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:56.998161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:56.998196 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:57.000222 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:57.000859 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:57.000863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:57.000864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:57.000967 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:57.001324 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:57.001327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:57.001329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:57.001582 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:57.002520 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.002693 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.002852 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10164,"thread_start_us":97,"threads_started":1}
I20260927 12:37:57.007791 13154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:37:57.012616 13154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:37:57.050952 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.037562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44461 (local address 127.0.0.1:49878)
0927 12:37:57.043098 (+ 5536us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.043167 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:57.043316 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:57.044386 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:57.044392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:57.044415 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:57.044562 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.044569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.046998 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.047001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.049097 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.049107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.050407 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.050411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.050516 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.050554 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 12:37:57.050656 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 12:37:57.050784 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5336,"mutex_wait_us":39,"thread_start_us":121,"threads_started":1}
I20260927 12:37:57.051568 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.038583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.0.98.1:44461)
0927 12:37:57.043609 (+ 5026us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:57.043617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:57.043640 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:37:57.043716 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:57.043722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:57.044010 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:57.044172 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:57.045643 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.046863 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.049219 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.050072 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.050176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.050348 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.051316 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.051431 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4910,"thread_start_us":115,"threads_started":1}
I20260927 12:37:57.055017 13275 tablet_service.cc:1467] Processing CreateTablet for tablet 433f1a5f41b14c4691901365cda43f8c (DEFAULT_TABLE table=test-table [id=7c0e12d4512b4ebaa1303f97440772b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:37:57.057314 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433f1a5f41b14c4691901365cda43f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:57.061882 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.040403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40557 (local address 127.0.0.1:40576)
0927 12:37:57.044243 (+ 3840us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.044256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:57.044340 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:57.045180 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:57.045183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:57.045198 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:57.045283 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.045289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.046998 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.047001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.050283 (+ 3282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.050293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.061407 (+ 11114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.061411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.061514 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.061538 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:37:57.061610 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:37:57.061717 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3773,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260927 12:37:57.062414 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.040802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.0.98.3:40557)
0927 12:37:57.044580 (+ 3778us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:57.044587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:57.044608 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:37:57.044680 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:57.044685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:57.044889 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:57.045025 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:57.045664 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.046863 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.060190 (+ 13327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.061067 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.061185 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.061342 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.062163 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.062274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3667,"thread_start_us":105,"threads_started":1}
I20260927 12:37:57.065497 13554 tablet_service.cc:1467] Processing CreateTablet for tablet 433f1a5f41b14c4691901365cda43f8c (DEFAULT_TABLE table=test-table [id=7c0e12d4512b4ebaa1303f97440772b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:37:57.066637 13554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433f1a5f41b14c4691901365cda43f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:57.071769 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.041224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.0.1:60346)
0927 12:37:57.045479 (+ 4255us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.045491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:57.045563 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:57.047634 (+ 2071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:57.047638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:57.047655 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:57.047749 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.047755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.068661 (+ 20906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.068665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.069884 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.069898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.071232 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.071236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.071347 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.071382 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 12:37:57.071444 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:37:57.071601 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4195,"thread_start_us":80,"threads_started":1}
I20260927 12:37:57.072515 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.041602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.98.2:38037)
0927 12:37:57.047243 (+ 5641us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:57.047249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:57.047268 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:37:57.047330 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:57.047333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:57.047496 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:57.047628 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:57.047875 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.068501 (+ 20626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.070057 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.071049 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.071997 (+ 948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.072205 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.072267 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.072373 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5535,"thread_start_us":92,"threads_started":1}
I20260927 12:37:57.075654 13657 tablet_bootstrap.cc:492] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Bootstrap starting.
I20260927 12:37:57.076198 13417 tablet_service.cc:1467] Processing CreateTablet for tablet 433f1a5f41b14c4691901365cda43f8c (DEFAULT_TABLE table=test-table [id=7c0e12d4512b4ebaa1303f97440772b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:37:57.077504 13417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 433f1a5f41b14c4691901365cda43f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:57.078135 13657 tablet_bootstrap.cc:654] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:57.079025 13657 log.cc:824] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:57.079389 13656 tablet_bootstrap.cc:492] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: Bootstrap starting.
I20260927 12:37:57.082549 13656 tablet_bootstrap.cc:654] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:57.084941 13656 log.cc:824] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:57.091851 13656 tablet_bootstrap.cc:492] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: No bootstrap required, opened a new log
I20260927 12:37:57.092129 13656 ts_tablet_manager.cc:1397] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260927 12:37:57.097745 13656 raft_consensus.cc:348] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.098135 13656 raft_consensus.cc:374] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:57.098371 13656 raft_consensus.cc:729] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4cab897f2b4563b510e9740ff4dc19, State: Initialized, Role: FOLLOWER
I20260927 12:37:57.098979 13656 consensus_queue.cc:260] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.100448 13656 ts_tablet_manager.cc:1428] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 12:37:57.100868 13352 heartbeater.cc:503] Master 127.0.98.62:34107 was elected leader, sending a full tablet report...
I20260927 12:37:57.106520 13657 tablet_bootstrap.cc:492] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: No bootstrap required, opened a new log
I20260927 12:37:57.106771 13657 ts_tablet_manager.cc:1397] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20260927 12:37:57.112509 13657 raft_consensus.cc:348] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.112919 13657 raft_consensus.cc:374] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:57.113008 13657 raft_consensus.cc:729] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a027f4463f42abb10a848f067d56f0, State: Initialized, Role: FOLLOWER
I20260927 12:37:57.113644 13657 consensus_queue.cc:260] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.114892 13657 ts_tablet_manager.cc:1428] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 12:37:57.115289 13620 heartbeater.cc:503] Master 127.0.98.62:34107 was elected leader, sending a full tablet report...
I20260927 12:37:57.117040 13660 tablet_bootstrap.cc:492] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Bootstrap starting.
I20260927 12:37:57.119716 13660 tablet_bootstrap.cc:654] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Neither blocks nor log segments found. Creating new log.
I20260927 12:37:57.120740 13660 log.cc:824] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Log is configured to *not* fsync() on all Append() calls
I20260927 12:37:57.129927 13660 tablet_bootstrap.cc:492] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: No bootstrap required, opened a new log
I20260927 12:37:57.130216 13660 ts_tablet_manager.cc:1397] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260927 12:37:57.134867 13660 raft_consensus.cc:348] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.135229 13660 raft_consensus.cc:374] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:37:57.135313 13660 raft_consensus.cc:729] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032c68861e1d411bbbd397c51917f0a9, State: Initialized, Role: FOLLOWER
I20260927 12:37:57.136085 13660 consensus_queue.cc:260] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.137415 13660 ts_tablet_manager.cc:1428] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 12:37:57.137854 13486 heartbeater.cc:503] Master 127.0.98.62:34107 was elected leader, sending a full tablet report...
W20260927 12:37:57.172528 13353 tablet.cc:2369] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:37:57.173730 13487 tablet.cc:2369] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:37:57.193118 13621 tablet.cc:2369] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:37:57.261767 13661 raft_consensus.cc:482] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:37:57.262007 13661 raft_consensus.cc:504] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.263355 13661 leader_election.cc:288] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 032c68861e1d411bbbd397c51917f0a9 (127.0.98.2:38037), c9a027f4463f42abb10a848f067d56f0 (127.0.98.3:40557)
I20260927 12:37:57.274782 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.264361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54119 (local address 127.0.98.2:38037)
0927 12:37:57.264511 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:57.264516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:57.266949 (+ 2433us) server_negotiation.cc:407] Connection header received
0927 12:37:57.267587 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:57.267591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:57.267652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:57.267703 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:57.268193 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.269004 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.273260 (+ 4256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.273911 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.273996 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.274070 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.274548 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.274638 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:37:57.275318 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.263656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.98.1:54119)
0927 12:37:57.266815 (+ 3159us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.266833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:57.267046 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:57.267853 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:57.267856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:57.267875 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:57.268018 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.268024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.271609 (+ 3585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.271614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.273061 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.273074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.274176 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.274180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.274393 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.274419 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:37:57.275039 (+ 620us) client_negotiation.cc:240] Negotiation successful
0927 12:37:57.275139 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3051,"mutex_wait_us":65,"thread_start_us":133,"threads_started":1}
I20260927 12:37:57.276002 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433f1a5f41b14c4691901365cda43f8c" candidate_uuid: "dc4cab897f2b4563b510e9740ff4dc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032c68861e1d411bbbd397c51917f0a9" is_pre_election: true
I20260927 12:37:57.276443 13421 raft_consensus.cc:2454] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4cab897f2b4563b510e9740ff4dc19 in term 0.
I20260927 12:37:57.277020 13239 leader_election.cc:302] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032c68861e1d411bbbd397c51917f0a9, dc4cab897f2b4563b510e9740ff4dc19; no voters:
I20260927 12:37:57.277410 13661 raft_consensus.cc:2781] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:37:57.277484 13661 raft_consensus.cc:482] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:37:57.277521 13661 raft_consensus.cc:3037] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:57.278888 13661 raft_consensus.cc:504] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.279165 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.264513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48487 (local address 127.0.98.3:40557)
0927 12:37:57.264637 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:57.264641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:57.270567 (+ 5926us) server_negotiation.cc:407] Connection header received
0927 12:37:57.272481 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:57.272485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:57.272540 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:57.272585 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:37:57.273001 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.273791 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.276682 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.277368 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.277451 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.277513 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.278927 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.279031 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 12:37:57.279453 13661 leader_election.cc:288] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [CANDIDATE]: Term 1 election: Requested vote from peers 032c68861e1d411bbbd397c51917f0a9 (127.0.98.2:38037), c9a027f4463f42abb10a848f067d56f0 (127.0.98.3:40557)
I20260927 12:37:57.279829 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.263771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40557 (local address 127.0.98.1:48487)
0927 12:37:57.270459 (+ 6688us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.270475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:57.272322 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:57.272734 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:57.272738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:57.272756 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:37:57.272878 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.272884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.273940 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.273943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.276283 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:57.276295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.277738 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.277742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:57.277848 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.277870 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:37:57.279568 (+ 1698us) client_negotiation.cc:240] Negotiation successful
0927 12:37:57.279670 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6604,"mutex_wait_us":764,"thread_start_us":110,"threads_started":1}
I20260927 12:37:57.280054 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433f1a5f41b14c4691901365cda43f8c" candidate_uuid: "dc4cab897f2b4563b510e9740ff4dc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "032c68861e1d411bbbd397c51917f0a9"
I20260927 12:37:57.280247 13421 raft_consensus.cc:3037] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:57.280781 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433f1a5f41b14c4691901365cda43f8c" candidate_uuid: "dc4cab897f2b4563b510e9740ff4dc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a027f4463f42abb10a848f067d56f0" is_pre_election: true
I20260927 12:37:57.281140 13574 raft_consensus.cc:2454] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4cab897f2b4563b510e9740ff4dc19 in term 0.
I20260927 12:37:57.281513 13573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "433f1a5f41b14c4691901365cda43f8c" candidate_uuid: "dc4cab897f2b4563b510e9740ff4dc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a027f4463f42abb10a848f067d56f0"
I20260927 12:37:57.281626 13573 raft_consensus.cc:3037] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:37:57.281658 13421 raft_consensus.cc:2454] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4cab897f2b4563b510e9740ff4dc19 in term 1.
I20260927 12:37:57.282917 13573 raft_consensus.cc:2454] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4cab897f2b4563b510e9740ff4dc19 in term 1.
I20260927 12:37:57.282971 13239 leader_election.cc:302] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032c68861e1d411bbbd397c51917f0a9, dc4cab897f2b4563b510e9740ff4dc19; no voters:
I20260927 12:37:57.283169 13661 raft_consensus.cc:2781] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:37:57.283394 13661 raft_consensus.cc:686] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [term 1 LEADER]: Becoming Leader. State: Replica: dc4cab897f2b4563b510e9740ff4dc19, State: Running, Role: LEADER
I20260927 12:37:57.283766 13661 consensus_queue.cc:237] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } }
I20260927 12:37:57.287902 13154 catalog_manager.cc:5495] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc4cab897f2b4563b510e9740ff4dc19 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dc4cab897f2b4563b510e9740ff4dc19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc4cab897f2b4563b510e9740ff4dc19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 } health_report { overall_health: UNKNOWN } } }
I20260927 12:37:57.349078 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.345392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.0.98.1:44461)
0927 12:37:57.345524 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:57.345528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:57.345544 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:37:57.345705 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:57.345708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:57.345767 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:57.345866 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:37:57.346127 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.346664 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.347495 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:57.347720 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:57.347807 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:57.348143 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:37:57.348671 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:37:57.348783 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:57.348848 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 12:37:57.348914 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:37:57.356415 13573 raft_consensus.cc:1261] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 1 FOLLOWER]: Refusing update from remote peer dc4cab897f2b4563b510e9740ff4dc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:37:57.356477 13421 raft_consensus.cc:1261] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 1 FOLLOWER]: Refusing update from remote peer dc4cab897f2b4563b510e9740ff4dc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:37:57.357343 13667 consensus_queue.cc:1035] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "032c68861e1d411bbbd397c51917f0a9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:37:57.357889 13661 consensus_queue.cc:1035] T 433f1a5f41b14c4691901365cda43f8c P dc4cab897f2b4563b510e9740ff4dc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a027f4463f42abb10a848f067d56f0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:37:57.369971 13673 mvcc.cc:205] Tried to move back new op lower bound from 7333939926443311104 to 7333939926158397440. Current Snapshot: MvccSnapshot[applied={T|T < 7333939926443311104}]
I20260927 12:37:57.372597 13677 mvcc.cc:205] Tried to move back new op lower bound from 7333939926443311104 to 7333939926158397440. Current Snapshot: MvccSnapshot[applied={T|T < 7333939926443311104}]
I20260927 12:37:57.381047 13679 mvcc.cc:205] Tried to move back new op lower bound from 7333939926443311104 to 7333939926158397440. Current Snapshot: MvccSnapshot[applied={T|T < 7333939926443311104}]
I20260927 12:37:57.389927 13154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:37:57.390136 13154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:37:57.393394 13154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4bf5d2f0f5e468f82414202a7afa3b8: Sending DeleteTablet for 3 replicas of tablet 433f1a5f41b14c4691901365cda43f8c
I20260927 12:37:57.394294 13275 tablet_service.cc:1514] Processing DeleteTablet for tablet 433f1a5f41b14c4691901365cda43f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:37:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49878
I20260927 12:37:57.395936 13417 tablet_service.cc:1514] Processing DeleteTablet for tablet 433f1a5f41b14c4691901365cda43f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:37:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60346
I20260927 12:37:57.397187 13554 tablet_service.cc:1514] Processing DeleteTablet for tablet 433f1a5f41b14c4691901365cda43f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:37:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40576
I20260927 12:37:57.398949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13192
I20260927 12:37:57.401643 13688 tablet_replica.cc:330] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: stopping tablet replica
I20260927 12:37:57.401988 13688 raft_consensus.cc:2229] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:37:57.402339 13688 raft_consensus.cc:2258] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:37:57.402767 13687 tablet_replica.cc:330] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: stopping tablet replica
I20260927 12:37:57.403103 13687 raft_consensus.cc:2229] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:37:57.403492 13687 raft_consensus.cc:2258] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:37:57.404505 13688 ts_tablet_manager.cc:1905] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:37:57.405032 13687 ts_tablet_manager.cc:1905] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:37:57.407908 13687 ts_tablet_manager.cc:1918] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:37:57.407908 13688 ts_tablet_manager.cc:1918] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:37:57.408028 13687 log.cc:1196] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/433f1a5f41b14c4691901365cda43f8c
I20260927 12:37:57.408228 13688 log.cc:1196] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/433f1a5f41b14c4691901365cda43f8c
I20260927 12:37:57.408488 13687 ts_tablet_manager.cc:1939] T 433f1a5f41b14c4691901365cda43f8c P 032c68861e1d411bbbd397c51917f0a9: Deleting consensus metadata
W20260927 12:37:57.409129 13138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44461 (error 108)
I20260927 12:37:57.409212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13355
I20260927 12:37:57.410429 13688 ts_tablet_manager.cc:1939] T 433f1a5f41b14c4691901365cda43f8c P c9a027f4463f42abb10a848f067d56f0: Deleting consensus metadata
I20260927 12:37:57.411729 13141 catalog_manager.cc:4841] TS c9a027f4463f42abb10a848f067d56f0 (127.0.98.3:40557): tablet 433f1a5f41b14c4691901365cda43f8c (table test-table [id=7c0e12d4512b4ebaa1303f97440772b3]) successfully deleted
I20260927 12:37:57.412494 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.412151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44461 (local address 127.0.0.1:49898)
0927 12:37:57.412340 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.412417 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 12:37:57.412693 13138 catalog_manager.cc:4584] TS dc4cab897f2b4563b510e9740ff4dc19 (127.0.98.1:44461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 433f1a5f41b14c4691901365cda43f8c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44461: connect: Connection refused (error 111)
I20260927 12:37:57.416018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13490
I20260927 12:37:57.416601 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:57.416246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38037 (local address 127.0.0.1:60352)
0927 12:37:57.416416 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:57.416465 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 12:37:57.421312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13121
I20260927 12:37:57.429829 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 12:37:57.765583 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 12:37:57.765758 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 12:37:58.101359 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 12:37:58.101 UTC [12188] LOG: received smart shutdown request
2026-09-27 12:37:58.106 UTC [12188] LOG: background worker "logical replication launcher" (PID 12197) exited with exit code 1
2026-09-27 12:37:58.106 UTC [12192] LOG: shutting down
2026-09-27 12:37:58.319 UTC [12188] LOG: database system is shut down
2026-09-27T12:37:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51363 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:37:58 dist-test-slave-tzgx krb5kdc[13698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:37:58 dist-test-slave-tzgx krb5kdc[13698](info): set up 2 sockets
Sep 27 12:37:58 dist-test-slave-tzgx krb5kdc[13698](info): commencing operation
krb5kdc: starting...
W20260927 12:37:59.396893 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:37:59.524955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37713
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37713,127.0.98.61:38793,127.0.98.60:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:37:59.649494 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:59.649794 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:59.649837 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:37:59.654373 13711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:37:59.654467 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:59.654487 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:59.654503 13711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:37:59.654518 13711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:37:59.660298 13711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37713,127.0.98.61:38793,127.0.98.60:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:59.661478 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:59.662499 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:37:59.671114 13719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:59.671115 13717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:59.671319 13711 server_base.cc:1034] running on GCE node
W20260927 12:37:59.671114 13716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:59.671980 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:59.673218 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:59.674417 13711 hybrid_clock.cc:648] HybridClock initialized: now 1790512679674398 us; error 32 us; skew 500 ppm
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:59.677472 13711 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:37:59.678435 13711 webserver.cc:460] Webserver started at http://127.0.98.62:38475/ using document root <none> and password file <none>
I20260927 12:37:59.678994 13711 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:59.679064 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:59.679276 13711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:37:59.681488 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "65cd811d74194e5e99628dd28a083644"
format_stamp: "Formatted at 2026-09-27 12:37:59 on dist-test-slave-tzgx"
server_key: "c3e25e4a956c3669745e45fd1b587f75"
server_key_iv: "852098e36267d209138a63dde2f7e338"
server_key_version: "encryptionkey@0"
I20260927 12:37:59.682062 13711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "65cd811d74194e5e99628dd28a083644"
format_stamp: "Formatted at 2026-09-27 12:37:59 on dist-test-slave-tzgx"
server_key: "c3e25e4a956c3669745e45fd1b587f75"
server_key_iv: "852098e36267d209138a63dde2f7e338"
server_key_version: "encryptionkey@0"
I20260927 12:37:59.686653 13711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:37:59.689438 13726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:59.690896 13711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:37:59.691046 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "65cd811d74194e5e99628dd28a083644"
format_stamp: "Formatted at 2026-09-27 12:37:59 on dist-test-slave-tzgx"
server_key: "c3e25e4a956c3669745e45fd1b587f75"
server_key_iv: "852098e36267d209138a63dde2f7e338"
server_key_version: "encryptionkey@0"
I20260927 12:37:59.691179 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:59.714265 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:59.717686 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:59.717880 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:59.725204 13711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37713
I20260927 12:37:59.725262 13778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37713 every 8 connection(s)
I20260927 12:37:59.726276 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:37:59.729878 13779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:59.732511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13711
I20260927 12:37:59.732752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:37:59.733076 392 external_mini_cluster.cc:1428] Setting key e9c87460bf461c435e746fd73172555f
I20260927 12:37:59.735723 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:37:59.753456 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.738307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.0.98.62:37713)
0927 12:37:59.739084 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:59.739097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:59.739151 (+ 54us) server_negotiation.cc:407] Connection header received
0927 12:37:59.740252 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:59.740259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:59.740558 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:59.740827 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:59.741733 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.742768 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.743942 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.744438 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.744612 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.747486 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:59.747493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:59.747509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:59.747536 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:59.749936 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.750507 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.750514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.750516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.750685 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.751078 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.751081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.751083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.751691 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:59.751832 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:59.752333 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 12:37:59.752693 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":636,"spinlock_wait_cycles":3840,"thread_start_us":176,"threads_started":1}
I20260927 12:37:59.753455 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.738306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37713 (local address 127.0.0.1:37126)
0927 12:37:59.738859 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.738917 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:59.739848 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:59.740856 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:59.740862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:59.741250 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:59.741544 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.741564 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.742908 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.742910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.743816 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.743827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.744651 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.744654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.744763 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.745398 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:59.745420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:59.747304 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:59.750092 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.750101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.750111 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.750382 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.750842 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.750845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.750847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.750979 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.751851 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:59.751857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:59.752007 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 12:37:59.752239 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 12:37:59.752594 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":139,"threads_started":1}
I20260927 12:37:59.756167 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:37:59.756953 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.756684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43436)
0927 12:37:59.756805 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.756881 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 12:37:59.757134 13730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
I20260927 12:37:59.757984 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.757764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43446)
0927 12:37:59.757870 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.757921 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 12:37:59.758109 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38793: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:37:59.764317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38793
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37713,127.0.98.61:38793,127.0.98.60:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:37:59.807431 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } attempt: 1
I20260927 12:37:59.808359 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.808024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43460)
0927 12:37:59.808213 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.808280 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 12:37:59.809065 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.808848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43468)
0927 12:37:59.808955 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.809003 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:37:59.809188 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38793: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
I20260927 12:37:59.877378 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } attempt: 2
W20260927 12:37:59.877357 13786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:37:59.877748 13786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:37:59.877825 13786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:37:59.878222 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.877934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43478)
0927 12:37:59.878081 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.878143 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 12:37:59.878916 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.878690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43484)
0927 12:37:59.878814 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.878857 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 12:37:59.879026 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38793: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38793: connect: Connection refused (error 111)
W20260927 12:37:59.882222 13786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:37:59.882289 13786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:37:59.882316 13786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:37:59.882350 13786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:37:59.882380 13786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:37:59.887917 13786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37713,127.0.98.61:38793,127.0.98.60:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:37:59.889115 13786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:37:59.890151 13786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:37:59.899147 13786 server_base.cc:1034] running on GCE node
W20260927 12:37:59.899099 13794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:59.899099 13791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:37:59.899099 13792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:37:59.899989 13786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:37:59.901242 13786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:37:59.902446 13786 hybrid_clock.cc:648] HybridClock initialized: now 1790512679902424 us; error 34 us; skew 500 ppm
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:37:59.905618 13786 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:37:59.906618 13786 webserver.cc:460] Webserver started at http://127.0.98.61:44489/ using document root <none> and password file <none>
I20260927 12:37:59.907164 13786 fs_manager.cc:359] Metadata directory not provided
I20260927 12:37:59.907227 13786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:37:59.907401 13786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:37:59.909806 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data/instance:
uuid: "79569d1fc7d0459ea044153381adf267"
format_stamp: "Formatted at 2026-09-27 12:37:59 on dist-test-slave-tzgx"
server_key: "a61deb8bc267cc616f35a98181dd75bf"
server_key_iv: "00fdb764780b91eb3babee72becac9f5"
server_key_version: "encryptionkey@0"
I20260927 12:37:59.910450 13786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal/instance:
uuid: "79569d1fc7d0459ea044153381adf267"
format_stamp: "Formatted at 2026-09-27 12:37:59 on dist-test-slave-tzgx"
server_key: "a61deb8bc267cc616f35a98181dd75bf"
server_key_iv: "00fdb764780b91eb3babee72becac9f5"
server_key_version: "encryptionkey@0"
I20260927 12:37:59.914535 13786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 12:37:59.917191 13801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:37:59.918924 13786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:37:59.919096 13786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal
uuid: "79569d1fc7d0459ea044153381adf267"
format_stamp: "Formatted at 2026-09-27 12:37:59 on dist-test-slave-tzgx"
server_key: "a61deb8bc267cc616f35a98181dd75bf"
server_key_iv: "00fdb764780b91eb3babee72becac9f5"
server_key_version: "encryptionkey@0"
I20260927 12:37:59.919251 13786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:37:59.944819 13786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:37:59.948125 13786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:37:59.948312 13786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:37:59.955438 13786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38793
I20260927 12:37:59.955565 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38793 every 8 connection(s)
I20260927 12:37:59.956465 13786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:37:59.959352 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:37:59.960791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13786
I20260927 12:37:59.960906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-1/wal/instance
I20260927 12:37:59.961180 392 external_mini_cluster.cc:1428] Setting key 8c37c1a1e84de64b451f83ababf75f95
I20260927 12:37:59.963981 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 12:37:59.970178 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:37:59.980507 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.970615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43498)
0927 12:37:59.970754 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.970769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:59.970856 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:59.971565 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:59.971567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:59.971770 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:59.971884 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.971889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.973429 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.973432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.973972 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.973984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.974765 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.974774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.974891 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.975378 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:59.975392 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:59.976760 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:59.978906 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.978909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.978911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.979087 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.979349 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.979352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.979354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.979436 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.979982 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:59.979985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:59.980074 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 12:37:59.980187 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 12:37:59.980295 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 12:37:59.980973 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.967017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37713 (local address 127.0.0.1:37140)
0927 12:37:59.967579 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.967622 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:59.968526 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:59.969050 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:59.969057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:59.969421 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:59.969918 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.969931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.970910 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.970914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.972168 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.972178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.972839 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.972843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.973017 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.973614 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:59.973635 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:59.975392 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:59.977319 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.977328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.977341 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.977708 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.978020 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.978023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.978025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.978255 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.978752 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:59.978760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:59.979135 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 12:37:59.979401 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 12:37:59.979764 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":164,"threads_started":1}
I20260927 12:37:59.981261 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.966431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.0.98.62:37713)
0927 12:37:59.966604 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:59.966607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:59.967735 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 12:37:59.968733 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:59.968737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:59.968801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:59.968869 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:59.970065 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.970788 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.972284 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.972679 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.972787 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.975532 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:59.975537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:59.975540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:59.975563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:59.977204 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.977802 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.977806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.977808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.977897 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.978384 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.978388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.978390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.978622 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:59.978738 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:59.980044 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0927 12:37:59.980212 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:37:59.981567 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.970715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.61:38793)
0927 12:37:59.971048 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:59.971060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:59.971082 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:37:59.971120 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:59.971125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:59.971331 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:59.971600 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:59.971996 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.973282 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.974112 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.974631 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.974731 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.976926 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:59.976934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:59.976951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:59.976977 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:59.978823 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.979186 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.979192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.979193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.979267 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.979539 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.979542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.979544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.979853 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:59.981183 (+ 1330us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:59.981292 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 12:37:59.981438 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":114,"threads_started":1}
I20260927 12:37:59.983557 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:37:59.983637 13779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:37:59.984400 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.984135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51608)
0927 12:37:59.984244 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.984310 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:37:59.985177 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.984903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51614)
0927 12:37:59.985068 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.985117 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 12:37:59.985337 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45273: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Sep 27 12:37:59 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:37:59.993201 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.984280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.0.98.61:38793)
0927 12:37:59.984387 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:37:59.984390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:37:59.984401 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:37:59.984456 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:37:59.984460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:37:59.984504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:37:59.984581 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:37:59.985069 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.985665 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.986509 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.986789 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.986906 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.989296 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:37:59.989299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:37:59.989301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:37:59.989327 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:37:59.991105 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.991652 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.991655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.991656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.991748 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:37:59.992082 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:37:59.992152 (+ 70us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:37:59.992155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:37:59.992396 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:37:59.992489 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:37:59.992899 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 12:37:59.993026 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:37:59.993186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45273
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37713,127.0.98.61:38793,127.0.98.60:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:37:59.993203 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.984122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43502)
0927 12:37:59.984244 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.984260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:37:59.984355 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:37:59.984675 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:37:59.984677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:37:59.984853 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:37:59.984942 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.984947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.985799 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.985802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.986380 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:37:59.986388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:37:59.986940 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:37:59.986943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:37:59.987030 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:37:59.987506 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:37:59.987518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:37:59.989126 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:37:59.991260 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.991264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.991265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.991510 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.991885 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:37:59.991888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:37:59.991889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:37:59.991990 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:37:59.992522 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:37:59.992525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:37:59.992686 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 12:37:59.992808 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 12:37:59.992958 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 12:37:59.995040 13854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:37:59.995879 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.995544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51626)
0927 12:37:59.995730 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.995809 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 12:37:59.996534 13802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
I20260927 12:37:59.997534 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:37:59.997319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51638)
0927 12:37:59.997429 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 12:37:59.997472 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:37:59.997704 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45273: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
I20260927 12:38:00.016512 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } attempt: 1
I20260927 12:38:00.017477 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.017137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51648)
0927 12:38:00.017317 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.017393 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 12:38:00.018327 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.018063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51662)
0927 12:38:00.018194 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.018257 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 12:38:00.018452 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45273: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
I20260927 12:38:00.046998 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } attempt: 1
I20260927 12:38:00.047880 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.047591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51670)
0927 12:38:00.047736 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.047798 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 12:38:00.048768 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.048520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51686)
0927 12:38:00.048651 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.048697 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 12:38:00.048919 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45273: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
I20260927 12:38:00.093644 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } attempt: 2
I20260927 12:38:00.094539 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.094244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51700)
0927 12:38:00.094395 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.094459 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 12:38:00.095309 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.095084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51712)
0927 12:38:00.095187 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.095249 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:38:00.095464 13779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45273: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
W20260927 12:38:00.112776 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:00.113078 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:00.113121 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:38:00.117125 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } attempt: 2
W20260927 12:38:00.117455 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:00.117507 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:00.117532 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:00.117556 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:00.117585 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:00.118003 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.117681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51728)
0927 12:38:00.117842 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.117911 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 12:38:00.118680 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.118453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51740)
0927 12:38:00.118549 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.118602 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 12:38:00.118806 13854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45273: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45273: connect: Connection refused (error 111)
I20260927 12:38:00.124133 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37713,127.0.98.61:38793,127.0.98.60:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:00.125344 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:00.126420 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:00.135807 13866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:00.135947 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:00.135895 13860 server_base.cc:1034] running on GCE node
W20260927 12:38:00.135811 13867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:00.136749 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:00.137948 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:00.139109 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790512680139096 us; error 45 us; skew 500 ppm
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:00.142838 13860 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:38:00.143887 13860 webserver.cc:460] Webserver started at http://127.0.98.60:34407/ using document root <none> and password file <none>
I20260927 12:38:00.144510 13860 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:00.144593 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:00.144812 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:00.147030 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data/instance:
uuid: "519b63bec70c4915a85e28e56d61d492"
format_stamp: "Formatted at 2026-09-27 12:38:00 on dist-test-slave-tzgx"
server_key: "24f488eff7f4cc3c954c0b52d0a68faf"
server_key_iv: "0f0339eb8daaf403eb2e2146ac626a3f"
server_key_version: "encryptionkey@0"
I20260927 12:38:00.147639 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal/instance:
uuid: "519b63bec70c4915a85e28e56d61d492"
format_stamp: "Formatted at 2026-09-27 12:38:00 on dist-test-slave-tzgx"
server_key: "24f488eff7f4cc3c954c0b52d0a68faf"
server_key_iv: "0f0339eb8daaf403eb2e2146ac626a3f"
server_key_version: "encryptionkey@0"
I20260927 12:38:00.151665 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:38:00.154523 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:00.156491 13860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:00.156649 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal
uuid: "519b63bec70c4915a85e28e56d61d492"
format_stamp: "Formatted at 2026-09-27 12:38:00 on dist-test-slave-tzgx"
server_key: "24f488eff7f4cc3c954c0b52d0a68faf"
server_key_iv: "0f0339eb8daaf403eb2e2146ac626a3f"
server_key_version: "encryptionkey@0"
I20260927 12:38:00.156798 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:00.173633 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:00.176980 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:00.177196 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:00.184499 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45273
I20260927 12:38:00.184516 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45273 every 8 connection(s)
I20260927 12:38:00.185530 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:38:00.188298 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:00.190106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 13860
I20260927 12:38:00.190202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512512765972-392-0/minicluster-data/master-2/wal/instance
I20260927 12:38:00.190471 392 external_mini_cluster.cc:1428] Setting key 0edea2c5dddee616bf662178fa8ca585
I20260927 12:38:00.192658 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:38:00.194835 13779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } attempt: 3
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:38:00.206369 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.195641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51764)
0927 12:38:00.195906 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.195918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:00.195996 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:00.196573 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:00.196576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:00.196747 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:00.196857 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.196865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.197843 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.197846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.198815 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.198826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.199395 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.199399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.199499 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.200446 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:00.200464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:00.201973 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:00.204392 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.204395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.204397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.204626 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.205029 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.205034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.205036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.205141 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.205858 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:00.205861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:00.205952 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:38:00.206031 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 12:38:00.206182 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":3}
I20260927 12:38:00.206553 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.195360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.0.98.62:37713)
0927 12:38:00.195828 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.195832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.195849 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:38:00.196020 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.196024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.196085 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.196167 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.197175 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.197879 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.199041 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.199352 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.199447 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.202630 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.202634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.202637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.202663 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.204215 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.205012 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.205016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.205017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.205092 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.205424 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.205427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.205429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.205691 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.205779 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.206238 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.206364 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":394}
I20260927 12:38:00.207861 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.195060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37713 (local address 127.0.0.1:37152)
0927 12:38:00.195700 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.195764 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:00.195896 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:00.196347 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:00.196353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:00.196835 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:00.197026 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.197049 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.197971 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.197974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.198935 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.198945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.199859 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.199866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.199978 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.200691 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:00.200712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:00.202469 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:00.204342 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.204351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.204363 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.204645 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.205174 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.205177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.205179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.205319 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.205854 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:00.205860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:00.205995 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 12:38:00.206170 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 12:38:00.206860 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":194,"threads_started":1}
I20260927 12:38:00.207870 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.191893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.98.60:45273)
0927 12:38:00.192363 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.192382 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.192428 (+ 46us) server_negotiation.cc:407] Connection header received
0927 12:38:00.193175 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.193201 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.193705 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.194106 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.195766 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.197352 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.198644 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.199236 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.199389 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.203266 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.203269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.203271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.203293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.205504 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.205943 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.205947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.205948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.206020 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.206315 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.206319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.206320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.206516 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.206718 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.207055 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.207161 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":316,"thread_start_us":159,"threads_started":1}
I20260927 12:38:00.207876 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.196021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.0.98.60:45273)
0927 12:38:00.196356 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.196365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.196379 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:38:00.196415 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.196417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.196458 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.196535 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.196955 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.197728 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.198930 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.199262 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.199930 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.202157 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.202167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.202186 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.202214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.204295 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.204736 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.204742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.204744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.204886 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.205235 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.205239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.205240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.205752 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.206758 (+ 1006us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.206863 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.206994 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260927 12:38:00.209944 13854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } attempt: 3
I20260927 12:38:00.210017 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:38:00.213152 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: Bootstrap starting.
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:38:00.216236 13779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: Neither blocks nor log segments found. Creating new log.
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:38:00.217089 13779 log.cc:824] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:00.221118 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.210398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51780)
0927 12:38:00.210674 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.210688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:00.210777 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:00.211593 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:00.211597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:00.211781 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:00.211886 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.211893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.212965 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.212968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.213691 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.213702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.214277 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.214281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.214359 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.215199 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:00.215214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:00.217061 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:00.219156 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.219160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.219163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.219613 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.219928 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.219932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.219934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.220024 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.220522 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:00.220524 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:00.220628 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 12:38:00.220798 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 12:38:00.220950 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260927 12:38:00.221410 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.210695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.0.98.61:38793)
0927 12:38:00.210806 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.210811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.211108 (+ 297us) server_negotiation.cc:407] Connection header received
0927 12:38:00.211232 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.211235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.211282 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.211352 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.211910 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.212697 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.213782 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.214117 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.214187 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.217735 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.217740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.217743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.217773 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.219415 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.220124 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.220127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.220129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.220212 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.220517 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.220520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.220522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.220738 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.220806 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.221158 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.221270 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":1408}
I20260927 12:38:00.221539 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.211055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.0.98.62:37713)
0927 12:38:00.211202 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.211206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.211216 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:38:00.211247 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.211249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.211283 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.211352 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.212187 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.212888 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.213923 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.214236 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.214301 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.218225 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.218229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.218232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.218259 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.219623 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.220152 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.220156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.220157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.220236 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.220535 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.220538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.220540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.220747 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.220892 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.221229 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.221355 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 12:38:00.221385 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: No bootstrap required, opened a new log
I20260927 12:38:00.221827 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.210468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38793 (local address 127.0.0.1:43510)
0927 12:38:00.211013 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.211024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:00.211136 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:00.211466 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:00.211469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:00.211657 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:00.211774 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.211781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.212824 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.212827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.213654 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.213664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.214345 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.214348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.214445 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.214946 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:00.214959 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:00.217566 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:00.219724 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.219727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.219729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.219983 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.220319 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.220322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.220324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.220407 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.220877 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:00.220880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:00.220971 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:38:00.221529 (+ 558us) client_negotiation.cc:240] Negotiation successful
0927 12:38:00.221683 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488}
I20260927 12:38:00.222609 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.210914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.0.98.60:45273)
0927 12:38:00.211273 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.211277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.211286 (+ 9us) server_negotiation.cc:407] Connection header received
0927 12:38:00.211312 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.211314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.211426 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.211997 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.212135 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.212841 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.213832 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.214155 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.215766 (+ 1611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.217242 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.217247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.217249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.217275 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.219023 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.219708 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.219712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.219714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.219818 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.220144 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.220147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.220149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.220393 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.220477 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.222366 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.222476 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260927 12:38:00.222959 13929 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:38:00.225855 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267: Bootstrap starting.
I20260927 12:38:00.225847 13779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.226092 13779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:00.226146 13779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65cd811d74194e5e99628dd28a083644, State: Initialized, Role: FOLLOWER
I20260927 12:38:00.226651 13779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.228081 13779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:00.228710 13854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267: Neither blocks nor log segments found. Creating new log.
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:38:00 dist-test-slave-tzgx krb5kdc[13698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:38:00.229661 13854 log.cc:824] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:00.232954 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267: No bootstrap required, opened a new log
I20260927 12:38:00.233588 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.223805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.98.60:45273)
0927 12:38:00.223898 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.223901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.224018 (+ 117us) server_negotiation.cc:407] Connection header received
0927 12:38:00.224182 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.224185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.224234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.224278 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.224777 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.225472 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.226474 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.226787 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.226856 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.229839 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.229843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.229846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.229872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.231868 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.232385 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.232388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.232390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.232467 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.232718 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.232721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.232722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.232937 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.233016 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.233300 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.233431 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 12:38:00.234045 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.223610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45273 (local address 127.0.0.1:51792)
0927 12:38:00.223964 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:00.223977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:00.224071 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:00.224437 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:00.224439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:00.224590 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:00.224675 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.224682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.225566 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.225569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.226377 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:00.226386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.226992 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.226995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:00.227070 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.227684 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:00.227699 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:00.229674 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:00.231988 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.231991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.231993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.232286 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.232559 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:00.232562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:00.232563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:00.232641 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:00.233055 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:00.233058 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:00.233138 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 12:38:00.233767 (+ 629us) client_negotiation.cc:240] Negotiation successful
0927 12:38:00.233900 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311}
W20260927 12:38:00.234015 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:38:00.234498 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:38:00.237241 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: Bootstrap starting.
I20260927 12:38:00.233548 13711 master_runner.cc:186] Error getting master registration for 127.0.98.61:38793: OK, instance_id { permanent_uuid: "79569d1fc7d0459ea044153381adf267" instance_seqno: 1790512679953388 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:38:00.237463 13711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:38:00.237385 13854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.237636 13854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:00.237696 13854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79569d1fc7d0459ea044153381adf267, State: Initialized, Role: FOLLOWER
I20260927 12:38:00.238320 13854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.239934 13854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:00.240363 13929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:00.241129 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.223016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.0.98.61:38793)
0927 12:38:00.223118 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.223122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.223300 (+ 178us) server_negotiation.cc:407] Connection header received
0927 12:38:00.223465 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.223468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.223511 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.223557 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.224505 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.225333 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.226474 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.226810 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.226870 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.229888 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.229892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.229895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.229919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.238261 (+ 8342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.238941 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.238945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.238946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.239038 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.239316 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.239319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.239321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.239673 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.239961 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.240852 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.240977 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:38:00.241297 13929 log.cc:824] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:00.242292 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.242519 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.244784 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.244948 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.245062 13950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:00.245141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:38:00.245191 13950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.246177 13950 leader_election.cc:288] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65cd811d74194e5e99628dd28a083644 (127.0.98.62:37713), 519b63bec70c4915a85e28e56d61d492 (127.0.98.60:45273)
I20260927 12:38:00.247690 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: No bootstrap required, opened a new log
I20260927 12:38:00.247687 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79569d1fc7d0459ea044153381adf267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cd811d74194e5e99628dd28a083644" is_pre_election: true
I20260927 12:38:00.248010 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79569d1fc7d0459ea044153381adf267 in term 0.
I20260927 12:38:00.248589 13805 leader_election.cc:302] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65cd811d74194e5e99628dd28a083644, 79569d1fc7d0459ea044153381adf267; no voters:
W20260927 12:38:00.250020 13963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:38:00.250120 13963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:38:00.252204 13929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.252493 13929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:00.252545 13929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519b63bec70c4915a85e28e56d61d492, State: Initialized, Role: FOLLOWER
I20260927 12:38:00.253111 13929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.250231 13786 master_runner.cc:186] Error getting master registration for 127.0.98.60:45273: OK, instance_id { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" instance_seqno: 1790512680182386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:38:00.254068 13786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:38:00.254258 13962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:38:00.254393 13962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:38:00.254494 13962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:00.254611 13929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:00.254652 13903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79569d1fc7d0459ea044153381adf267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519b63bec70c4915a85e28e56d61d492" is_pre_election: true
I20260927 12:38:00.254922 13903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79569d1fc7d0459ea044153381adf267 in term 0.
I20260927 12:38:00.256930 13962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.257813 13903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79569d1fc7d0459ea044153381adf267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519b63bec70c4915a85e28e56d61d492"
I20260927 12:38:00.257827 13753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79569d1fc7d0459ea044153381adf267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65cd811d74194e5e99628dd28a083644"
I20260927 12:38:00.257929 13903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:00.257947 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:00.259146 13903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79569d1fc7d0459ea044153381adf267 in term 1.
I20260927 12:38:00.259155 13753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79569d1fc7d0459ea044153381adf267 in term 1.
I20260927 12:38:00.260257 13802 leader_election.cc:302] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519b63bec70c4915a85e28e56d61d492, 79569d1fc7d0459ea044153381adf267; no voters:
I20260927 12:38:00.260476 13950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:00.260740 13950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [term 1 LEADER]: Becoming Leader. State: Replica: 79569d1fc7d0459ea044153381adf267, State: Running, Role: LEADER
I20260927 12:38:00.261143 13950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } }
I20260927 12:38:00.262840 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79569d1fc7d0459ea044153381adf267. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.262969 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:00.264348 13962 leader_election.cc:288] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [CANDIDATE]: Term 1 election: Requested vote from peers 65cd811d74194e5e99628dd28a083644 (127.0.98.62:37713), 519b63bec70c4915a85e28e56d61d492 (127.0.98.60:45273)
I20260927 12:38:00.272432 13983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:00.273455 13983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:00.274694 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.264957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.98.60:45273)
0927 12:38:00.265071 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.265076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.265824 (+ 748us) server_negotiation.cc:407] Connection header received
0927 12:38:00.266010 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.266013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.266063 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.266112 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.267067 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.267795 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.268950 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.269273 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.269342 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.271053 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.271056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.271059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.271084 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.272716 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.273244 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.273248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.273249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.273335 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.273721 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.273724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.273726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.273951 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.274050 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.274405 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.274526 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:38:00.275314 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.277756 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.282311 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.266021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.0.98.62:37713)
0927 12:38:00.266134 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.266149 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.267204 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 12:38:00.267365 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.267368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.267413 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.267459 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.268441 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.269160 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.270477 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.270839 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.270915 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.277947 (+ 7032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.277952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.277955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.277988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.279874 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.280406 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.280410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.280412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.280502 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.281023 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.281027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.281029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.281498 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.281635 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.281990 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.282122 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:38:00.284183 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:00.266639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.0.98.61:38793)
0927 12:38:00.266748 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:00.266752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:00.275530 (+ 8778us) server_negotiation.cc:407] Connection header received
0927 12:38:00.275725 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:00.275728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:00.275781 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:00.275832 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:00.276687 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.277455 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.278602 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:00.278953 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:00.279034 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:00.280499 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:00.280504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:00.280506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:00.280532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:00.282034 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.282814 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.282818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.282820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.282914 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:00.283206 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:00.283210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:00.283212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:00.283416 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:00.283520 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:00.283862 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 12:38:00.283987 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:38:00.285431 13903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [term 1 FOLLOWER]: Refusing update from remote peer 79569d1fc7d0459ea044153381adf267: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 12:38:00.285658 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:38:00.285741 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:38:00.286358 13981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:00.287437 13753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [term 1 FOLLOWER]: Refusing update from remote peer 79569d1fc7d0459ea044153381adf267: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:00.288003 13981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:00.291571 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79569d1fc7d0459ea044153381adf267. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.291716 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.291860 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79569d1fc7d0459ea044153381adf267. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.293865 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.299460 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.299623 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:00.300645 13988 mvcc.cc:205] Tried to move back new op lower bound from 7333939938434723840 to 7333939938352742400. Current Snapshot: MvccSnapshot[applied={T|T < 7333939938434723840}]
I20260927 12:38:00.302155 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.302285 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:00.304502 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.304647 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.305181 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79569d1fc7d0459ea044153381adf267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65cd811d74194e5e99628dd28a083644" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37713 } } peers { permanent_uuid: "79569d1fc7d0459ea044153381adf267" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38793 } } peers { permanent_uuid: "519b63bec70c4915a85e28e56d61d492" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45273 } } }
I20260927 12:38:00.305305 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:38:00.305516 13991 mvcc.cc:205] Tried to move back new op lower bound from 7333939938434723840 to 7333939938352742400. Current Snapshot: MvccSnapshot[applied={T|T < 7333939938434723840}]
I20260927 12:38:00.306092 13989 mvcc.cc:205] Tried to move back new op lower bound from 7333939938434723840 to 7333939938352742400. Current Snapshot: MvccSnapshot[applied={T|T < 7333939938434723840}]
I20260927 12:38:00.316774 13983 catalog_manager.cc:1347] Generated new cluster ID: 3f9d1d3e88a24734a1073eca76030ba1
I20260927 12:38:00.316888 13983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:38:00.349720 13983 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:38:00.351095 13983 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:38:00.372598 13983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79569d1fc7d0459ea044153381adf267: Generated new TSK 0
I20260927 12:38:00.373560 13983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:38:01.235973 13948 catalog_manager.cc:1259] Loaded cluster ID: 3f9d1d3e88a24734a1073eca76030ba1
I20260927 12:38:01.236063 13948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: loading cluster ID for follower catalog manager: success
I20260927 12:38:01.237926 13948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: acquiring CA information for follower catalog manager: success
I20260927 12:38:01.239280 13948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65cd811d74194e5e99628dd28a083644: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:38:01.287050 13980 catalog_manager.cc:1259] Loaded cluster ID: 3f9d1d3e88a24734a1073eca76030ba1
I20260927 12:38:01.287150 13980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: loading cluster ID for follower catalog manager: success
I20260927 12:38:01.288861 13980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: acquiring CA information for follower catalog manager: success
I20260927 12:38:01.290035 13980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 519b63bec70c4915a85e28e56d61d492: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:38:01.443308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13711
I20260927 12:38:01.448619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13786
I20260927 12:38:01.453748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 13860
2026-09-27T12:38:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3040 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:38:01 dist-test-slave-tzgx krb5kdc[14002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:38:01 dist-test-slave-tzgx krb5kdc[14002](info): set up 2 sockets
Sep 27 12:38:01 dist-test-slave-tzgx krb5kdc[14002](info): commencing operation
krb5kdc: starting...
W20260927 12:38:02.432348 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:38:02 dist-test-slave-tzgx krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:38:02.560570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:02.676633 14015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:02.676936 14015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:02.676980 14015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:02.681257 14015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:02.681334 14015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:02.681353 14015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:02.681368 14015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:02.681383 14015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:02.686975 14015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:02.688344 14015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:02.689462 14015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:02.699342 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:02.699342 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:02.699385 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:02.699630 14015 server_base.cc:1034] running on GCE node
I20260927 12:38:02.700341 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:02.701562 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:02.702766 14015 hybrid_clock.cc:648] HybridClock initialized: now 1790512682702746 us; error 45 us; skew 500 ppm
Sep 27 12:38:02 dist-test-slave-tzgx krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:02.705745 14015 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:38:02.706764 14015 webserver.cc:460] Webserver started at http://127.0.98.62:44803/ using document root <none> and password file <none>
I20260927 12:38:02.707334 14015 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:02.707402 14015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:02.707597 14015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:02.709746 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "2c8a00b97c8b443c9ad8797e6a14b89f"
format_stamp: "Formatted at 2026-09-27 12:38:02 on dist-test-slave-tzgx"
server_key: "057cb6cc05130e04a8dff1c4dd364a28"
server_key_iv: "bf09cc3547028b25a2a4f7026dab2727"
server_key_version: "encryptionkey@0"
I20260927 12:38:02.710304 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c8a00b97c8b443c9ad8797e6a14b89f"
format_stamp: "Formatted at 2026-09-27 12:38:02 on dist-test-slave-tzgx"
server_key: "057cb6cc05130e04a8dff1c4dd364a28"
server_key_iv: "bf09cc3547028b25a2a4f7026dab2727"
server_key_version: "encryptionkey@0"
I20260927 12:38:02.714529 14015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:38:02.717540 14030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:02.718959 14015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:38:02.719096 14015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "2c8a00b97c8b443c9ad8797e6a14b89f"
format_stamp: "Formatted at 2026-09-27 12:38:02 on dist-test-slave-tzgx"
server_key: "057cb6cc05130e04a8dff1c4dd364a28"
server_key_iv: "bf09cc3547028b25a2a4f7026dab2727"
server_key_version: "encryptionkey@0"
I20260927 12:38:02.719237 14015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:02.730738 14015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:02.734163 14015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:02.734331 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:02.741153 14015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43657
I20260927 12:38:02.741176 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43657 every 8 connection(s)
I20260927 12:38:02.742154 14015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:38:02.745193 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:02.747480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14015
I20260927 12:38:02.747607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:38:02.747879 392 external_mini_cluster.cc:1428] Setting key 2f569ce62f39242e82f5dbeef71c6002
I20260927 12:38:02.751948 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f: Bootstrap starting.
I20260927 12:38:02.754402 14083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:02.755184 14083 log.cc:824] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:38:02 dist-test-slave-tzgx krb5kdc[14002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:02.757783 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f: No bootstrap required, opened a new log
I20260927 12:38:02.761955 14083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43657 } }
I20260927 12:38:02.762285 14083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:02.762343 14083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8a00b97c8b443c9ad8797e6a14b89f, State: Initialized, Role: FOLLOWER
I20260927 12:38:02.763005 14083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43657 } }
I20260927 12:38:02.763247 14083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:38:02.763327 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:38:02.763429 14083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:02.764627 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:02.749433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.0.98.62:43657)
0927 12:38:02.750053 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:02.750069 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:02.750108 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:38:02.750955 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:02.750974 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:02.751354 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:02.751701 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:02.752578 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:02.753595 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:02.754485 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:02.755045 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:02.755241 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:02.758107 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:02.758124 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:02.758145 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:02.758184 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:02.760874 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:02.761412 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:02.761419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:02.761426 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:02.761581 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:02.761897 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:02.761900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:02.761902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:02.762465 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:02.762667 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:02.763055 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 12:38:02.763430 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":247,"threads_started":1}
I20260927 12:38:02.764853 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43657 } }
I20260927 12:38:02.765247 14083 leader_election.cc:302] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c8a00b97c8b443c9ad8797e6a14b89f; no voters:
I20260927 12:38:02.765650 14083 leader_election.cc:288] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:38:02.765756 14088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:02.765990 14088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [term 1 LEADER]: Becoming Leader. State: Replica: 2c8a00b97c8b443c9ad8797e6a14b89f, State: Running, Role: LEADER
I20260927 12:38:02.766336 14088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43657 } }
I20260927 12:38:02.766993 14083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:02.768077 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43657 } } }
I20260927 12:38:02.768356 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:02.769150 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:38:02.769022 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c8a00b97c8b443c9ad8797e6a14b89f. Latest consensus state: current_term: 1 leader_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a00b97c8b443c9ad8797e6a14b89f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43657 } } }
I20260927 12:38:02.770370 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f [sys.catalog]: This master's current role is: LEADER
W20260927 12:38:02.773280 14104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:38:02.773355 14104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:38:02.773423 14092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:02.774080 14092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:02.779443 14092 catalog_manager.cc:1347] Generated new cluster ID: 9005ac24b7724c32a0aed553d670637c
I20260927 12:38:02.779532 14092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:38:02.790047 14092 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:38:02.791038 14092 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:38:02.800812 14092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c8a00b97c8b443c9ad8797e6a14b89f: Generated new TSK 0
I20260927 12:38:02.801548 14092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:38:02.825839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14015
2026-09-27T12:38:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1370 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:38:02 dist-test-slave-tzgx krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:38:02 dist-test-slave-tzgx krb5kdc[14110](info): set up 2 sockets
Sep 27 12:38:02 dist-test-slave-tzgx krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260927 12:38:03.796314 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:38:03 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:38:03.923897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34781
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:04.042987 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:04.043308 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:04.043352 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:04.048071 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:04.048208 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:04.048233 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:04.048251 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:04.048269 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:04.055306 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:04.056667 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:04.057754 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:04.067914 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.068562 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.068565 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:04.068543 14123 server_base.cc:1034] running on GCE node
I20260927 12:38:04.069904 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:04.071376 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:04.072561 14123 hybrid_clock.cc:648] HybridClock initialized: now 1790512684072542 us; error 35 us; skew 500 ppm
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:04.080029 14123 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:38:04.081203 14123 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260927 12:38:04.081782 14123 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:04.081853 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:04.082056 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:04.084272 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "26fb8700a7a1418488ccb98d8877fd93"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "384ca87ca275c2a77e9e9692c7eaac06"
server_key_iv: "7db49652a2b04f30fd9ffd714013b5af"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.084859 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "26fb8700a7a1418488ccb98d8877fd93"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "384ca87ca275c2a77e9e9692c7eaac06"
server_key_iv: "7db49652a2b04f30fd9ffd714013b5af"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.088963 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:38:04.091619 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.093047 14123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:04.093220 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "26fb8700a7a1418488ccb98d8877fd93"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "384ca87ca275c2a77e9e9692c7eaac06"
server_key_iv: "7db49652a2b04f30fd9ffd714013b5af"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.093343 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:04.122665 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:04.128300 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:04.128712 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:04.137887 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34781
I20260927 12:38:04.138190 14190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34781 every 8 connection(s)
I20260927 12:38:04.139257 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:38:04.142242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14123
I20260927 12:38:04.142349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:38:04.142524 14191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:04.142617 392 external_mini_cluster.cc:1428] Setting key 12668256885fe88d54b4bcb8edc0862c
I20260927 12:38:04.149739 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap starting.
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:04.152355 14191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:04.153188 14191 log.cc:824] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:04.156018 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: No bootstrap required, opened a new log
I20260927 12:38:04.159373 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.144174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.0.98.62:34781)
0927 12:38:04.144676 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.144692 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.144733 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:38:04.145494 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.145515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.146204 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.146742 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:04.147562 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.148687 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.149584 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.150123 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.150289 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.153056 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:04.153064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:04.153086 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:04.153126 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:04.155703 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.156295 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.156302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.156309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.156466 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.156798 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.156802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.156803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.157341 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:04.157533 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.157829 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.158156 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":146,"threads_started":1}
I20260927 12:38:04.161386 14191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:04.161809 14191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:04.161899 14191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fb8700a7a1418488ccb98d8877fd93, State: Initialized, Role: FOLLOWER
I20260927 12:38:04.162422 14191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:04.162599 14191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:38:04.162667 14191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:38:04.162766 14191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:04.163802 14191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:04.164250 14191 leader_election.cc:302] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26fb8700a7a1418488ccb98d8877fd93; no voters:
I20260927 12:38:04.164597 14191 leader_election.cc:288] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:38:04.164757 14196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:04.164978 14196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 LEADER]: Becoming Leader. State: Replica: 26fb8700a7a1418488ccb98d8877fd93, State: Running, Role: LEADER
I20260927 12:38:04.165342 14196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:04.165949 14191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:04.166952 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26fb8700a7a1418488ccb98d8877fd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } } }
I20260927 12:38:04.167112 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:04.167513 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26fb8700a7a1418488ccb98d8877fd93. Latest consensus state: current_term: 1 leader_uuid: "26fb8700a7a1418488ccb98d8877fd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } } }
I20260927 12:38:04.167634 14205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:04.167677 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:04.171527 14205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:04.178629 14205 catalog_manager.cc:1347] Generated new cluster ID: cd641c3f981846d799e96873feb5e965
I20260927 12:38:04.178719 14205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:38:04.198568 14205 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:38:04.199805 14205 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:38:04.206604 14205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Generated new TSK 0
I20260927 12:38:04.207427 14205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:38:04.239596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34781
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:04.356732 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:04.357008 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:04.357053 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:04.361543 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:04.361636 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:04.361701 14217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:38:04.367539 14217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:04.368685 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:04.370280 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:04.380643 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:04.380743 14217 server_base.cc:1034] running on GCE node
W20260927 12:38:04.380664 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.381101 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:04.381585 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:04.382236 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:04.383441 14217 hybrid_clock.cc:648] HybridClock initialized: now 1790512684383424 us; error 33 us; skew 500 ppm
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:04.386245 14217 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:38:04.387137 14217 webserver.cc:460] Webserver started at http://127.0.98.1:33111/ using document root <none> and password file <none>
I20260927 12:38:04.387670 14217 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:04.387728 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:04.387892 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:04.389986 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "731b811510b14b7781f67a07f98f00a0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "cb0b971baecf768895b42aca060e3c92"
server_key_iv: "ce280ae59a3db4f2fed8808120c3f6a5"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.390506 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "731b811510b14b7781f67a07f98f00a0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "cb0b971baecf768895b42aca060e3c92"
server_key_iv: "ce280ae59a3db4f2fed8808120c3f6a5"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.394312 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:38:04.396905 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.398350 14217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:04.398473 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "731b811510b14b7781f67a07f98f00a0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "cb0b971baecf768895b42aca060e3c92"
server_key_iv: "ce280ae59a3db4f2fed8808120c3f6a5"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.398586 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:04.426537 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:04.430541 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:04.430877 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:04.431558 14217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:04.432543 14217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:04.432596 14217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.432654 14217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:04.432683 14217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.441812 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35817
I20260927 12:38:04.441962 14345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35817 every 8 connection(s)
I20260927 12:38:04.442806 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:38:04.447067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14217
I20260927 12:38:04.447211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:38:04.447479 392 external_mini_cluster.cc:1428] Setting key e121bd3184e55ca2bf9e00e02c2416b8
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:38:04.458357 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.444801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51455 (local address 127.0.98.62:34781)
0927 12:38:04.444985 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.444990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.446314 (+ 1324us) server_negotiation.cc:407] Connection header received
0927 12:38:04.447296 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.447300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.447355 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.447399 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:04.449169 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.449930 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.451198 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.451444 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.451536 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.454196 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:04.454200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:04.454202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:04.454228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:04.455849 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.456677 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.456681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.456683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.456769 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.457127 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.457129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.457130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.457362 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:04.457453 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.458030 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.458187 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:38:04.459254 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.445247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:51455)
0927 12:38:04.446110 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:04.446180 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:04.447068 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:04.447779 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:04.447789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:04.448294 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:04.448995 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.449015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.450077 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.450080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.451057 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.451069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.451601 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.451606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.451817 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.452468 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:04.452486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:04.453984 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:04.456085 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:04.456129 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:04.456151 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:04.456524 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:04.456859 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:04.456863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:04.456865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:04.457021 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:04.457474 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:04.457482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:04.457776 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 12:38:04.457955 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 12:38:04.458323 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":144,"threads_started":1}
I20260927 12:38:04.460620 14346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34781
I20260927 12:38:04.460868 14346 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:04.461369 14346 heartbeater.cc:511] Master 127.0.98.62:34781 requested a full tablet report, sending...
I20260927 12:38:04.462966 14151 ts_manager.cc:194] Registered new tserver with Master: 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817)
I20260927 12:38:04.464170 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:38:04.476790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34781
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:04.591504 14351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:04.591805 14351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:04.591849 14351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:04.596206 14351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:04.596282 14351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:04.596349 14351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:38:04.602207 14351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:04.603369 14351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:04.604429 14351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:04.614717 14359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.615029 14357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.615042 14356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:04.615741 14351 server_base.cc:1034] running on GCE node
I20260927 12:38:04.616328 14351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:04.617081 14351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:04.618238 14351 hybrid_clock.cc:648] HybridClock initialized: now 1790512684618206 us; error 46 us; skew 500 ppm
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:04.621171 14351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:38:04.622148 14351 webserver.cc:460] Webserver started at http://127.0.98.2:37135/ using document root <none> and password file <none>
I20260927 12:38:04.622738 14351 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:04.622808 14351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:04.623018 14351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:04.625494 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "3f633fd0d7f6ee1b9268491c9386bdf1"
server_key_iv: "89a3dd5541e1893c9dc771b1e9887839"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.626113 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "3f633fd0d7f6ee1b9268491c9386bdf1"
server_key_iv: "89a3dd5541e1893c9dc771b1e9887839"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.630268 14351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:38:04.632970 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.634327 14351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:04.634459 14351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "3f633fd0d7f6ee1b9268491c9386bdf1"
server_key_iv: "89a3dd5541e1893c9dc771b1e9887839"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.634586 14351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:04.648257 14351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:04.652242 14351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:04.652431 14351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:04.653091 14351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:04.653973 14351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:04.654022 14351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.654076 14351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:04.654105 14351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.663589 14351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38009
I20260927 12:38:04.663599 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38009 every 8 connection(s)
I20260927 12:38:04.664608 14351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:38:04.673607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14351
I20260927 12:38:04.673725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:38:04.673972 392 external_mini_cluster.cc:1428] Setting key 154915fafddcc431b8426336b9ac97db
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:04.678686 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.666571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47403 (local address 127.0.98.62:34781)
0927 12:38:04.666765 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.666770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.667738 (+ 968us) server_negotiation.cc:407] Connection header received
0927 12:38:04.669140 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.669143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.669204 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.669278 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:04.670522 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.671091 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.672508 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.672757 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.672851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.675382 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:04.675386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:04.675389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:04.675415 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:04.676713 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.677215 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.677218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.677220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.677295 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.677613 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.677616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.677618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.677804 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:04.677887 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.678398 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.678524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:38:04.679615 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.667042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:47403)
0927 12:38:04.667574 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:04.667624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:04.668913 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:04.669459 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:04.669467 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:04.669858 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:04.670376 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.670391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.671234 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.671237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.672382 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.672394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.672918 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.672923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.673094 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.673695 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:04.673712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:04.675201 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:04.676830 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:04.676837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:04.676857 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:04.677120 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:04.677375 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:04.677378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:04.677380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:04.677527 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:04.677901 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:04.677908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:04.678171 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 12:38:04.678344 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 12:38:04.678701 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260927 12:38:04.680770 14480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34781
I20260927 12:38:04.681058 14480 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:04.681730 14480 heartbeater.cc:511] Master 127.0.98.62:34781 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:38:04.682986 14151 ts_manager.cc:194] Registered new tserver with Master: b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
I20260927 12:38:04.683552 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:38:04.703894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34781
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:04.815785 14485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:04.816077 14485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:04.816162 14485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:04.820393 14485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:04.820466 14485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:04.820528 14485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:38:04.826095 14485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:04.827142 14485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:04.828195 14485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:04.838085 14491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.838101 14490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:04.838080 14493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:04.838271 14485 server_base.cc:1034] running on GCE node
I20260927 12:38:04.838918 14485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:04.839578 14485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:04.840754 14485 hybrid_clock.cc:648] HybridClock initialized: now 1790512684840733 us; error 33 us; skew 500 ppm
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:04.843421 14485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:38:04.844398 14485 webserver.cc:460] Webserver started at http://127.0.98.3:42899/ using document root <none> and password file <none>
I20260927 12:38:04.844976 14485 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:04.845047 14485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:04.845247 14485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:04.847328 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "eeea41cea85b454797ef2a082341b3c0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "15cff7a97f3b10ddaccdf61812e6b47e"
server_key_iv: "cf5d71c3ef6241eba74798ccd74620e8"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.847882 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eeea41cea85b454797ef2a082341b3c0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "15cff7a97f3b10ddaccdf61812e6b47e"
server_key_iv: "cf5d71c3ef6241eba74798ccd74620e8"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.851735 14485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:38:04.854393 14500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.855785 14485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:04.855917 14485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "eeea41cea85b454797ef2a082341b3c0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "15cff7a97f3b10ddaccdf61812e6b47e"
server_key_iv: "cf5d71c3ef6241eba74798ccd74620e8"
server_key_version: "encryptionkey@0"
I20260927 12:38:04.856024 14485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:04.878551 14485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:04.881912 14485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:04.882154 14485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:04.882805 14485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:04.883675 14485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:04.883723 14485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.883776 14485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:04.883806 14485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:04.892437 14485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41389
I20260927 12:38:04.892511 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41389 every 8 connection(s)
I20260927 12:38:04.893391 14485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:38:04.900604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14485
I20260927 12:38:04.900729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:38:04.901023 392 external_mini_cluster.cc:1428] Setting key 3fe5dd8355113af786e7dc3238cc9e54
Sep 27 12:38:04 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:04.907593 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.895377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50517 (local address 127.0.98.62:34781)
0927 12:38:04.895525 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.895530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.896525 (+ 995us) server_negotiation.cc:407] Connection header received
0927 12:38:04.897804 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.897807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.897854 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.897929 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:04.899072 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.899621 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.900908 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.901226 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.901342 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.904013 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:04.904017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:04.904020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:04.904048 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:04.905682 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.906257 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.906260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.906261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.906340 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.906615 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.906618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.906620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.906801 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:04.906874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.907361 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.907467 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:38:04.908581 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.895808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:50517)
0927 12:38:04.896356 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:04.896417 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:04.897585 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:04.898083 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:04.898090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:04.898469 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:04.898944 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.898957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.899735 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.899737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.900789 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.900800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.901392 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.901396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.901591 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.902230 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:04.902262 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:04.903853 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:04.905812 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:04.905820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:04.905840 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:04.906164 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:04.906410 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:04.906412 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:04.906414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:04.906546 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:04.906877 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:04.906882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:04.907140 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 12:38:04.907304 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:38:04.907585 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":113,"threads_started":1}
I20260927 12:38:04.909785 14614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34781
I20260927 12:38:04.910014 14614 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:04.910503 14614 heartbeater.cc:511] Master 127.0.98.62:34781 requested a full tablet report, sending...
I20260927 12:38:04.911639 14151 ts_manager.cc:194] Registered new tserver with Master: eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389)
I20260927 12:38:04.912317 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50517
I20260927 12:38:04.914841 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:38:04.924230 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.916943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.62:34781)
0927 12:38:04.917053 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.917058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.917261 (+ 203us) server_negotiation.cc:407] Connection header received
0927 12:38:04.917365 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.917367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.917409 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.917476 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:04.918102 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.918625 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.919438 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.919683 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.919771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.921117 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:04.921121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:04.921123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:04.921147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:04.922423 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.922926 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.922929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.922930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.923018 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:04.923351 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:04.923355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:04.923356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:04.923592 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:04.923705 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.923946 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.924048 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 12:38:04.927266 14151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:38:04.930408 14151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:38:04.945631 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.940605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35817 (local address 127.0.0.1:38878)
0927 12:38:04.940864 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:04.940877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:04.940986 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:04.941523 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:04.941526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:04.941541 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:04.941632 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.941638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.943068 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.943071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.944077 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.944112 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.945012 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.945016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.945154 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.945197 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 12:38:04.945315 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 12:38:04.945454 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260927 12:38:04.945760 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.939462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.0.98.2:38009)
0927 12:38:04.940289 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.940303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.940327 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:38:04.940404 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.940409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.940626 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.941148 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:04.941337 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.942452 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.944273 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.945137 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.945293 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.945412 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.945497 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.945602 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":116,"threads_started":1}
I20260927 12:38:04.946112 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.940338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.0.98.1:35817)
0927 12:38:04.940766 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.940784 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.940982 (+ 198us) server_negotiation.cc:407] Connection header received
0927 12:38:04.941091 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.941096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.941280 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.941420 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:04.941747 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.942935 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.944220 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.944887 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.944994 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.945223 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.945857 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.945964 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":81,"threads_started":1}
I20260927 12:38:04.945636 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.939367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38009 (local address 127.0.0.1:34368)
0927 12:38:04.939763 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:04.939819 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:04.939955 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:04.940802 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:04.940808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:04.940842 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:04.940999 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.941007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.942579 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.942590 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.944144 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.944151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.945296 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.945300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.945381 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.945401 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:38:04.945465 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:38:04.945523 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":79,"spinlock_wait_cycles":3968,"thread_start_us":104,"threads_started":1}
I20260927 12:38:04.946411 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.940241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.0.98.3:41389)
0927 12:38:04.940951 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:04.940962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:04.940984 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:38:04.941057 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:04.941062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:04.941213 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:04.941342 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:04.941579 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.942373 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.944326 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.945137 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.945247 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.945391 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:04.946168 (+ 777us) server_negotiation.cc:299] Negotiation successful
0927 12:38:04.946275 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":152,"threads_started":1}
I20260927 12:38:04.945852 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:04.939738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.0.1:37080)
0927 12:38:04.940745 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:04.940758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:04.940842 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:04.941336 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:04.941339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:04.941355 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:04.941458 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.941464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.942505 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.942508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.944180 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:04.944193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:04.945456 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:04.945459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:04.945556 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:04.945578 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:38:04.945645 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:38:04.945725 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":178,"spinlock_wait_cycles":6400,"thread_start_us":100,"threads_started":1}
I20260927 12:38:04.949185 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb684669174e158047ae7db1a0347f (DEFAULT_TABLE table=test-table [id=282bbc9a96ac4a62a0190683c16f161e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:04.949332 14548 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb684669174e158047ae7db1a0347f (DEFAULT_TABLE table=test-table [id=282bbc9a96ac4a62a0190683c16f161e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:04.949564 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecb684669174e158047ae7db1a0347f (DEFAULT_TABLE table=test-table [id=282bbc9a96ac4a62a0190683c16f161e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:04.950229 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb684669174e158047ae7db1a0347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:04.950397 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb684669174e158047ae7db1a0347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:04.950405 14548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecb684669174e158047ae7db1a0347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:04.956580 14631 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Bootstrap starting.
I20260927 12:38:04.958074 14633 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Bootstrap starting.
I20260927 12:38:04.958173 14632 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Bootstrap starting.
I20260927 12:38:04.960033 14631 tablet_bootstrap.cc:654] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:04.960268 14633 tablet_bootstrap.cc:654] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:04.960490 14632 tablet_bootstrap.cc:654] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:04.960891 14631 log.cc:824] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:04.961090 14633 log.cc:824] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:04.961553 14632 log.cc:824] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:04.963001 14631 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: No bootstrap required, opened a new log
I20260927 12:38:04.963214 14631 ts_tablet_manager.cc:1397] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260927 12:38:04.963400 14632 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: No bootstrap required, opened a new log
I20260927 12:38:04.963564 14632 ts_tablet_manager.cc:1397] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:38:04.963764 14633 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: No bootstrap required, opened a new log
I20260927 12:38:04.963938 14633 ts_tablet_manager.cc:1397] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:38:04.967078 14633 raft_consensus.cc:348] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:04.967423 14633 raft_consensus.cc:374] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:04.967491 14633 raft_consensus.cc:729] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731b811510b14b7781f67a07f98f00a0, State: Initialized, Role: FOLLOWER
I20260927 12:38:04.967368 14632 raft_consensus.cc:348] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:04.967368 14631 raft_consensus.cc:348] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:04.967651 14631 raft_consensus.cc:374] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:04.967729 14631 raft_consensus.cc:729] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bd7c240fb34d43a4dc1aa7e65fa735, State: Initialized, Role: FOLLOWER
I20260927 12:38:04.967675 14632 raft_consensus.cc:374] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:04.967793 14632 raft_consensus.cc:729] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeea41cea85b454797ef2a082341b3c0, State: Initialized, Role: FOLLOWER
I20260927 12:38:04.967965 14633 consensus_queue.cc:260] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:04.968223 14632 consensus_queue.cc:260] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:04.968225 14631 consensus_queue.cc:260] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:04.968658 14346 heartbeater.cc:503] Master 127.0.98.62:34781 was elected leader, sending a full tablet report...
I20260927 12:38:04.968973 14633 ts_tablet_manager.cc:1428] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 12:38:04.969182 14631 ts_tablet_manager.cc:1428] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:38:04.969543 14480 heartbeater.cc:503] Master 127.0.98.62:34781 was elected leader, sending a full tablet report...
I20260927 12:38:04.969545 14614 heartbeater.cc:503] Master 127.0.98.62:34781 was elected leader, sending a full tablet report...
I20260927 12:38:04.969734 14632 ts_tablet_manager.cc:1428] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:38:05.087004 14638 raft_consensus.cc:482] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:05.087250 14638 raft_consensus.cc:504] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:05.088497 14638 leader_election.cc:288] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817), eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389)
I20260927 12:38:05.093518 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.089101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39117 (local address 127.0.98.3:41389)
0927 12:38:05.089224 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.089228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.089241 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:38:05.089361 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.089365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.089412 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.089478 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:05.089742 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.090473 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.092237 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.092890 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.092961 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.093025 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.093319 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.093395 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:38:05.093545 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.088905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41149 (local address 127.0.98.1:35817)
0927 12:38:05.089038 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.089043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.089183 (+ 140us) server_negotiation.cc:407] Connection header received
0927 12:38:05.089251 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.089254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.089311 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.089398 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:05.089669 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.090471 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.092236 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.092860 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.092962 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.093025 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.093351 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.093426 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:38:05.093545 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.088807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35817 (local address 127.0.98.2:41149)
0927 12:38:05.089020 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.089035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:05.089157 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:05.089424 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:05.089426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:05.089441 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:05.089562 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.089569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.090608 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.090611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.092062 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.092074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.093085 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.093088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.093223 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.093244 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:38:05.093318 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:38:05.093399 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":76}
I20260927 12:38:05.093554 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.088883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.98.2:39117)
0927 12:38:05.089185 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.089197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:05.089266 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:05.089498 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:05.089501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:05.089512 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:05.089605 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.089643 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.090603 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.090635 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.092062 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.092074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.093029 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.093032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.093170 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.093215 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 12:38:05.093280 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:38:05.093363 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":108,"thread_start_us":135,"threads_started":1}
I20260927 12:38:05.094280 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb684669174e158047ae7db1a0347f" candidate_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731b811510b14b7781f67a07f98f00a0" is_pre_election: true
I20260927 12:38:05.094323 14568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb684669174e158047ae7db1a0347f" candidate_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeea41cea85b454797ef2a082341b3c0" is_pre_election: true
I20260927 12:38:05.094630 14568 raft_consensus.cc:2454] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6bd7c240fb34d43a4dc1aa7e65fa735 in term 0.
I20260927 12:38:05.094626 14300 raft_consensus.cc:2454] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6bd7c240fb34d43a4dc1aa7e65fa735 in term 0.
I20260927 12:38:05.095126 14368 leader_election.cc:302] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0, b6bd7c240fb34d43a4dc1aa7e65fa735; no voters:
I20260927 12:38:05.095381 14638 raft_consensus.cc:2781] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:38:05.095464 14638 raft_consensus.cc:482] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:38:05.095513 14638 raft_consensus.cc:3037] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:05.096572 14638 raft_consensus.cc:504] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:05.096904 14638 leader_election.cc:288] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [CANDIDATE]: Term 1 election: Requested vote from peers 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817), eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389)
I20260927 12:38:05.097322 14568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb684669174e158047ae7db1a0347f" candidate_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeea41cea85b454797ef2a082341b3c0"
I20260927 12:38:05.097322 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecb684669174e158047ae7db1a0347f" candidate_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731b811510b14b7781f67a07f98f00a0"
I20260927 12:38:05.097457 14300 raft_consensus.cc:3037] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:05.097457 14568 raft_consensus.cc:3037] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:05.098716 14568 raft_consensus.cc:2454] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6bd7c240fb34d43a4dc1aa7e65fa735 in term 1.
I20260927 12:38:05.098716 14300 raft_consensus.cc:2454] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6bd7c240fb34d43a4dc1aa7e65fa735 in term 1.
I20260927 12:38:05.098996 14368 leader_election.cc:302] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0, b6bd7c240fb34d43a4dc1aa7e65fa735; no voters:
I20260927 12:38:05.099160 14638 raft_consensus.cc:2781] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:05.099380 14638 raft_consensus.cc:686] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 LEADER]: Becoming Leader. State: Replica: b6bd7c240fb34d43a4dc1aa7e65fa735, State: Running, Role: LEADER
I20260927 12:38:05.099725 14638 consensus_queue.cc:237] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:05.103111 14155 catalog_manager.cc:5495] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } health_report { overall_health: UNKNOWN } } }
I20260927 12:38:05.138257 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.134686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.0.98.2:38009)
0927 12:38:05.134810 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.134815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.134828 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:38:05.134958 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.134960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.135013 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.135099 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:38:05.135346 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.135841 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.136735 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.136947 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.137034 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.137331 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:38:05.137853 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:38:05.137975 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.138047 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.138110 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:38:05.144316 14615 tablet.cc:2369] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:38:05.144583 14568 raft_consensus.cc:1261] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Refusing update from remote peer b6bd7c240fb34d43a4dc1aa7e65fa735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:05.144583 14300 raft_consensus.cc:1261] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Refusing update from remote peer b6bd7c240fb34d43a4dc1aa7e65fa735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:05.145349 14638 consensus_queue.cc:1035] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:05.145537 14641 consensus_queue.cc:1035] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:05.153730 14644 mvcc.cc:205] Tried to move back new op lower bound from 7333939958344462336 to 7333939958171197440. Current Snapshot: MvccSnapshot[applied={T|T < 7333939958344462336}]
I20260927 12:38:05.154276 14645 mvcc.cc:205] Tried to move back new op lower bound from 7333939958344462336 to 7333939958171197440. Current Snapshot: MvccSnapshot[applied={T|T < 7333939958344462336}]
I20260927 12:38:05.165014 14151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:05.165225 14151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:05.168044 14151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Sending DeleteTablet for 3 replicas of tablet 9ecb684669174e158047ae7db1a0347f
I20260927 12:38:05.168999 14548 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb684669174e158047ae7db1a0347f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37080
I20260927 12:38:05.169000 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb684669174e158047ae7db1a0347f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34368
I20260927 12:38:05.169154 14280 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb684669174e158047ae7db1a0347f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38878
I20260927 12:38:05.169538 14656 tablet_replica.cc:330] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: stopping tablet replica
I20260927 12:38:05.169598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14217
I20260927 12:38:05.172359 14654 tablet_replica.cc:330] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: stopping tablet replica
I20260927 12:38:05.172712 14654 raft_consensus.cc:2229] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:05.173089 14654 raft_consensus.cc:2258] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:05.173494 14655 tablet_replica.cc:330] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: stopping tablet replica
I20260927 12:38:05.173852 14655 raft_consensus.cc:2229] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:38:05.174337 14655 raft_consensus.cc:2258] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:05.174899 14654 ts_tablet_manager.cc:1905] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:38:05.176023 14140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35817 (error 108)
I20260927 12:38:05.176208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14351
I20260927 12:38:05.178143 14654 ts_tablet_manager.cc:1918] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:05.178169 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.177777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35817 (local address 127.0.0.1:38888)
0927 12:38:05.177994 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.178075 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 12:38:05.178253 14654 log.cc:1196] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/9ecb684669174e158047ae7db1a0347f
W20260927 12:38:05.178318 14140 catalog_manager.cc:4584] TS 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ecb684669174e158047ae7db1a0347f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35817: connect: Connection refused (error 111)
I20260927 12:38:05.178586 14654 ts_tablet_manager.cc:1939] T 9ecb684669174e158047ae7db1a0347f P eeea41cea85b454797ef2a082341b3c0: Deleting consensus metadata
I20260927 12:38:05.179735 14139 catalog_manager.cc:4841] TS eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): tablet 9ecb684669174e158047ae7db1a0347f (table test-table [id=282bbc9a96ac4a62a0190683c16f161e]) successfully deleted
I20260927 12:38:05.182171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14485
I20260927 12:38:05.184044 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.183799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38009 (local address 127.0.0.1:34386)
0927 12:38:05.183916 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.183971 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 12:38:05.186980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14123
I20260927 12:38:05.192473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34781
--webserver_interface=127.0.98.62
--webserver_port=40545
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:05.308305 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:05.308615 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:05.308674 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:05.313009 14658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:05.313097 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:05.313123 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:05.313144 14658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:05.313166 14658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:05.318900 14658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:05.320055 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:05.321182 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:05.334939 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.334941 14665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.334921 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:05.335134 14658 server_base.cc:1034] running on GCE node
I20260927 12:38:05.335857 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:05.337124 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:05.338328 14658 hybrid_clock.cc:648] HybridClock initialized: now 1790512685338305 us; error 38 us; skew 500 ppm
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:05.341383 14658 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:38:05.342507 14658 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260927 12:38:05.343127 14658 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:05.343217 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:05.347532 14658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:05.350155 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:05.351555 14658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:05.351699 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "26fb8700a7a1418488ccb98d8877fd93"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "384ca87ca275c2a77e9e9692c7eaac06"
server_key_iv: "7db49652a2b04f30fd9ffd714013b5af"
server_key_version: "encryptionkey@0"
I20260927 12:38:05.352403 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:05.365305 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:05.368644 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:05.368844 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:05.376610 14658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34781
I20260927 12:38:05.376948 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34781 every 8 connection(s)
I20260927 12:38:05.377821 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:38:05.379205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14658
I20260927 12:38:05.379696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35817
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34781
--webserver_port=33111
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:05.383159 14727 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:38:05.385075 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap starting.
I20260927 12:38:05.407821 14727 log.cc:824] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:05.419076 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:05.419492 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap complete.
I20260927 12:38:05.424861 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:05.425487 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fb8700a7a1418488ccb98d8877fd93, State: Initialized, Role: FOLLOWER
I20260927 12:38:05.426111 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:05.426326 14727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:38:05.426416 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:38:05.426545 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:38:05.428802 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:05.429210 14727 leader_election.cc:302] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26fb8700a7a1418488ccb98d8877fd93; no voters:
I20260927 12:38:05.429575 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:38:05.429729 14731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:38:05.430001 14731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 LEADER]: Becoming Leader. State: Replica: 26fb8700a7a1418488ccb98d8877fd93, State: Running, Role: LEADER
I20260927 12:38:05.430332 14731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:05.430949 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:05.431736 14732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "26fb8700a7a1418488ccb98d8877fd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } } }
I20260927 12:38:05.431865 14732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:05.432296 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26fb8700a7a1418488ccb98d8877fd93. Latest consensus state: current_term: 2 leader_uuid: "26fb8700a7a1418488ccb98d8877fd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } } }
I20260927 12:38:05.432438 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:05.436611 14746 catalog_manager.cc:1259] Loaded cluster ID: cd641c3f981846d799e96873feb5e965
I20260927 12:38:05.436712 14746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: loading cluster ID for follower catalog manager: success
W20260927 12:38:05.437820 14746 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260927 12:38:05.437902 14746 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260927 12:38:05.437980 14739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:05.440421 14739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:05.440925 14739 catalog_manager.cc:1259] Loaded cluster ID: cd641c3f981846d799e96873feb5e965
I20260927 12:38:05.440982 14739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:38:05.442009 14739 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 12:38:05.446585 14739 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 12:38:05.452705 14739 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:38:05.453596 14739 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Loaded TSK: 0
I20260927 12:38:05.454924 14739 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:38:05.516952 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:05.517226 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:05.517272 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:05.521402 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:05.521478 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:05.521539 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:38:05.527060 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33111
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:05.528203 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:05.529220 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:05.539347 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.539436 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:05.539544 14729 server_base.cc:1034] running on GCE node
W20260927 12:38:05.539358 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:05.540268 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:05.541005 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:05.542215 14729 hybrid_clock.cc:648] HybridClock initialized: now 1790512685542176 us; error 46 us; skew 500 ppm
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:05.545264 14729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:38:05.546290 14729 webserver.cc:460] Webserver started at http://127.0.98.1:33111/ using document root <none> and password file <none>
I20260927 12:38:05.546952 14729 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:05.547043 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:05.551244 14729 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:05.553675 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:05.555127 14729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:38:05.555266 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "731b811510b14b7781f67a07f98f00a0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "cb0b971baecf768895b42aca060e3c92"
server_key_iv: "ce280ae59a3db4f2fed8808120c3f6a5"
server_key_version: "encryptionkey@0"
I20260927 12:38:05.555912 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:05.575099 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:05.578667 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:05.578860 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:05.579473 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:05.580718 14771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:38:05.586076 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:38:05.586174 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 12:38:05.586246 14729 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:38:05.588079 14729 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:38:05.588176 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 12:38:05.588488 14771 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Bootstrap starting.
I20260927 12:38:05.598009 14729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35817
I20260927 12:38:05.598098 14878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35817 every 8 connection(s)
I20260927 12:38:05.599018 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:38:05.606920 14771 log.cc:824] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:05.607074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14729
I20260927 12:38:05.607539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38009
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34781
--webserver_port=37135
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:05.615855 14771 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:05.616323 14771 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Bootstrap complete.
I20260927 12:38:05.617527 14771 ts_tablet_manager.cc:1397] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.005s
I20260927 12:38:05.619215 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.601140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46165 (local address 127.0.98.62:34781)
0927 12:38:05.601519 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.601538 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.601873 (+ 335us) server_negotiation.cc:407] Connection header received
0927 12:38:05.603046 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.603067 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.603586 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.603996 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:05.605179 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.606617 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.608444 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.609180 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.609372 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.612628 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:05.612637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:05.612657 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:05.612709 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:05.615058 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.615801 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.615809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.615816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.616024 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.616592 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.616596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.616598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.617091 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:05.617236 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.618006 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.618349 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260927 12:38:05.619228 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.601017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:46165)
0927 12:38:05.601673 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.601729 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:05.602556 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:05.604074 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:05.604117 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:05.604510 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:05.604992 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.605008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.606851 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.606855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.608248 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.608262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.609439 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.609444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.609673 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.610418 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:05.610440 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:05.612353 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:05.615250 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.615256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.615271 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.615642 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.616239 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.616243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.616245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.616458 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.617237 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:05.617245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:05.617603 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 12:38:05.617838 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 12:38:05.618233 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":147,"threads_started":1}
I20260927 12:38:05.620708 14771 raft_consensus.cc:348] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:05.621217 14771 raft_consensus.cc:729] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731b811510b14b7781f67a07f98f00a0, State: Initialized, Role: FOLLOWER
I20260927 12:38:05.621735 14771 consensus_queue.cc:260] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:05.622439 14879 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34781
I20260927 12:38:05.622733 14879 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:05.622879 14771 ts_tablet_manager.cc:1428] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:38:05.623286 14879 heartbeater.cc:511] Master 127.0.98.62:34781 requested a full tablet report, sending...
I20260927 12:38:05.625972 14691 ts_manager.cc:194] Registered new tserver with Master: 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817)
I20260927 12:38:05.629709 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46165
I20260927 12:38:05.632427 14879 heartbeater.cc:503] Master 127.0.98.62:34781 was elected leader, sending a full tablet report...
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:38:05.642282 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.629131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35817 (local address 127.0.0.1:38898)
0927 12:38:05.629968 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.630023 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:05.630198 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:05.630788 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:05.630794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:05.631262 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:05.631411 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.631420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.634125 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.634129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.635093 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.635106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.635743 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.635766 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.635890 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.636523 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:05.636538 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:05.638172 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:05.640272 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.640285 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.640291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.640625 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.640917 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.640921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.640922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.641034 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.641572 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:05.641578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:05.641720 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 12:38:05.641864 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 12:38:05.642082 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":118,"threads_started":1}
I20260927 12:38:05.642330 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.629240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.0.98.1:35817)
0927 12:38:05.629688 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.629700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.630147 (+ 447us) server_negotiation.cc:407] Connection header received
0927 12:38:05.630392 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.630398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.630620 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.630780 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:05.632922 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.633940 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.635251 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.635635 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.635736 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.638383 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:05.638396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:05.638411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:05.638441 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:05.640138 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.640744 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.640749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.640751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.640833 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.641175 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.641178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.641180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.641463 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:05.641573 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.642011 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.642185 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":120,"threads_started":1}
I20260927 12:38:05.643529 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb684669174e158047ae7db1a0347f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38898
I20260927 12:38:05.644448 14888 tablet_replica.cc:330] stopping tablet replica
I20260927 12:38:05.644750 14888 raft_consensus.cc:2229] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:05.644982 14888 raft_consensus.cc:2258] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:05.645977 14888 ts_tablet_manager.cc:1905] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:05.649609 14888 ts_tablet_manager.cc:1918] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:05.649712 14888 log.cc:1196] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/9ecb684669174e158047ae7db1a0347f
I20260927 12:38:05.649955 14888 ts_tablet_manager.cc:1939] T 9ecb684669174e158047ae7db1a0347f P 731b811510b14b7781f67a07f98f00a0: Deleting consensus metadata
I20260927 12:38:05.651697 14676 catalog_manager.cc:4841] TS 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817): tablet 9ecb684669174e158047ae7db1a0347f (table test-table [id=282bbc9a96ac4a62a0190683c16f161e]) successfully deleted
W20260927 12:38:05.738663 14884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:05.738999 14884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:05.739073 14884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:05.743417 14884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:05.743518 14884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:05.743623 14884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:38:05.749459 14884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37135
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:05.750608 14884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:05.751662 14884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:05.763195 14893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.763191 14894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.763199 14896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:05.763543 14884 server_base.cc:1034] running on GCE node
I20260927 12:38:05.763973 14884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:05.764700 14884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:05.765877 14884 hybrid_clock.cc:648] HybridClock initialized: now 1790512685765859 us; error 32 us; skew 500 ppm
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:05.768744 14884 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:38:05.769658 14884 webserver.cc:460] Webserver started at http://127.0.98.2:37135/ using document root <none> and password file <none>
I20260927 12:38:05.770290 14884 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:05.770376 14884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:05.774482 14884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:38:05.776857 14903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:05.778388 14884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:38:05.778532 14884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "3f633fd0d7f6ee1b9268491c9386bdf1"
server_key_iv: "89a3dd5541e1893c9dc771b1e9887839"
server_key_version: "encryptionkey@0"
I20260927 12:38:05.779110 14884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:05.790087 14884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:05.793493 14884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:05.793748 14884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:05.794382 14884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:05.795538 14910 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:38:05.799669 14884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:38:05.799757 14884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 12:38:05.799834 14884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:38:05.801668 14884 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:38:05.801717 14884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 12:38:05.802033 14910 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Bootstrap starting.
I20260927 12:38:05.808569 14910 log.cc:824] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:05.811287 14884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38009
I20260927 12:38:05.811408 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38009 every 8 connection(s)
I20260927 12:38:05.812278 14884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:38:05.815613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 14884
I20260927 12:38:05.815977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41389
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34781
--webserver_port=42899
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:05.818686 14910 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:05.819157 14910 tablet_bootstrap.cc:492] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Bootstrap complete.
I20260927 12:38:05.819532 14910 ts_tablet_manager.cc:1397] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260927 12:38:05.823057 14910 raft_consensus.cc:348] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
I20260927 12:38:05.823678 14910 raft_consensus.cc:729] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bd7c240fb34d43a4dc1aa7e65fa735, State: Initialized, Role: FOLLOWER
I20260927 12:38:05.824545 14910 consensus_queue.cc:260] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } }
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:05.825553 14910 ts_tablet_manager.cc:1428] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 12:38:05.830677 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.814356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55045 (local address 127.0.98.62:34781)
0927 12:38:05.814520 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.814526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.815092 (+ 566us) server_negotiation.cc:407] Connection header received
0927 12:38:05.817472 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.817477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.817548 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.817604 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:05.819413 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.820458 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.822213 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.822586 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.822727 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.826401 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:05.826406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:05.826410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:05.826450 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:05.828083 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.828812 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.828815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.828817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.828909 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.829268 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.829271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.829273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.829503 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:05.829597 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.830349 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.830503 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:38:05.831844 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.814400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:55045)
0927 12:38:05.814910 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.814963 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:05.816459 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:05.818004 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:05.818013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:05.818531 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:05.819213 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.819236 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.820638 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.820642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.822022 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.822035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.822883 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.822888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.823115 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.824167 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:05.824189 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:05.826104 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:05.828275 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.828283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.828302 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.828708 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.828992 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.828994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.828995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.829183 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.829659 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:05.829665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:05.830085 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 12:38:05.830286 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 12:38:05.830755 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260927 12:38:05.833215 15019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34781
I20260927 12:38:05.833528 15019 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:05.834156 15019 heartbeater.cc:511] Master 127.0.98.62:34781 requested a full tablet report, sending...
I20260927 12:38:05.836404 14691 ts_manager.cc:194] Registered new tserver with Master: b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
I20260927 12:38:05.837728 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55045
I20260927 12:38:05.840400 15019 heartbeater.cc:503] Master 127.0.98.62:34781 was elected leader, sending a full tablet report...
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 12:38:05.846616 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.837462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38009 (local address 127.0.0.1:34396)
0927 12:38:05.837664 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:05.837679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:05.837768 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:05.838612 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:05.838616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:05.838832 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:05.838923 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.838929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.839946 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.839950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.840569 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:05.840582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.841262 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.841279 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:05.841385 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.841940 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:05.841952 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:05.843134 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:05.845051 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.845054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.845056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.845256 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.845537 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:05.845540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:05.845542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:05.845623 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:05.846074 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:05.846077 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:05.846181 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 12:38:05.846274 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 12:38:05.846442 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 12:38:05.846783 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:05.837607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.0.98.2:38009)
0927 12:38:05.838101 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:05.838113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:05.838136 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:38:05.838231 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:05.838237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:05.838471 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:05.838608 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:05.839041 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.839795 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.840684 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:05.841143 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:05.841250 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:05.843271 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:05.843281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:05.843292 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:05.843314 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:05.844935 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.845364 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.845369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.845371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.845450 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:05.845707 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:05.845710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:05.845712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:05.845970 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:05.846091 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:05.846371 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 12:38:05.846541 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":391,"thread_start_us":138,"threads_started":1}
I20260927 12:38:05.847791 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecb684669174e158047ae7db1a0347f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34396
I20260927 12:38:05.848985 15025 tablet_replica.cc:330] stopping tablet replica
I20260927 12:38:05.849306 15025 raft_consensus.cc:2229] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:05.849630 15025 raft_consensus.cc:2258] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:05.851478 15025 ts_tablet_manager.cc:1905] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:05.856370 15025 ts_tablet_manager.cc:1918] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:05.856482 15025 log.cc:1196] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/9ecb684669174e158047ae7db1a0347f
I20260927 12:38:05.856721 15025 ts_tablet_manager.cc:1939] T 9ecb684669174e158047ae7db1a0347f P b6bd7c240fb34d43a4dc1aa7e65fa735: Deleting consensus metadata
I20260927 12:38:05.859130 14676 catalog_manager.cc:4841] TS b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): tablet 9ecb684669174e158047ae7db1a0347f (table test-table [id=282bbc9a96ac4a62a0190683c16f161e]) successfully deleted
W20260927 12:38:05.949287 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:05.949923 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:05.949987 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:05.954852 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:05.954936 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:05.955000 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:38:05.961443 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42899
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:05.962601 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:05.963608 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:05.973433 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.973433 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:05.973496 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:05.973773 15022 server_base.cc:1034] running on GCE node
I20260927 12:38:05.974206 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:05.974895 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:05.976081 15022 hybrid_clock.cc:648] HybridClock initialized: now 1790512685976057 us; error 40 us; skew 500 ppm
Sep 27 12:38:05 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:05.978999 15022 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:38:05.979935 15022 webserver.cc:460] Webserver started at http://127.0.98.3:42899/ using document root <none> and password file <none>
I20260927 12:38:05.980508 15022 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:05.980592 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:05.984786 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:38:05.987036 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:05.988734 15022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:05.988870 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "eeea41cea85b454797ef2a082341b3c0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "15cff7a97f3b10ddaccdf61812e6b47e"
server_key_iv: "cf5d71c3ef6241eba74798ccd74620e8"
server_key_version: "encryptionkey@0"
I20260927 12:38:05.989454 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:06.034541 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:06.037784 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:06.037976 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:06.038578 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:06.039899 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:06.039959 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:06.040021 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:06.040053 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:06.049350 15022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41389
I20260927 12:38:06.049513 15153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41389 every 8 connection(s)
I20260927 12:38:06.050446 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:38:06.053968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15022
Sep 27 12:38:06 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:06.064237 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.054651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.0.98.62:34781)
0927 12:38:06.054872 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.054877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.055114 (+ 237us) server_negotiation.cc:407] Connection header received
0927 12:38:06.055252 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.055254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.055286 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.055354 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:06.056451 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.057349 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.058592 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.058853 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.058934 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.060719 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:06.060724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:06.060727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:06.060759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:06.062306 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:06.062846 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:06.062851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:06.062852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:06.062956 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:06.063271 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:06.063275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:06.063277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:06.063528 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:06.063736 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.063928 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.064036 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
I20260927 12:38:06.066447 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.052678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52681 (local address 127.0.98.62:34781)
0927 12:38:06.052817 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.052820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.053526 (+ 706us) server_negotiation.cc:407] Connection header received
0927 12:38:06.054929 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.054932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.054977 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.055108 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:06.056720 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.057417 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.059002 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.059213 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.059298 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.062334 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:06.062337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:06.062340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:06.062365 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:06.063766 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:06.064443 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:06.064446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:06.064448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:06.064531 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:06.064955 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:06.064958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:06.064960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:06.065272 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:06.065389 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.066140 (+ 751us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.066276 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:38:06.067248 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.052856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:52681)
0927 12:38:06.053364 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.053409 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:06.054681 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:06.055355 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:06.055365 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:06.055887 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:06.056576 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.056592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.057554 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.057558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.058876 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.058887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.059365 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.059369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.059558 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.060203 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:06.060226 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:06.062143 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:06.063875 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:06.063882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:06.063901 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:06.064325 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:06.064628 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:06.064631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:06.064633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:06.064844 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:06.065412 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:06.065420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:06.065820 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 12:38:06.066039 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 12:38:06.066401 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260927 12:38:06.068460 15154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34781
I20260927 12:38:06.068678 15154 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:06.069130 15154 heartbeater.cc:511] Master 127.0.98.62:34781 requested a full tablet report, sending...
I20260927 12:38:06.070091 14691 ts_manager.cc:194] Registered new tserver with Master: eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389)
I20260927 12:38:06.070648 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52681
I20260927 12:38:06.078282 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:38:06.082973 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.078957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.0.98.62:34781)
0927 12:38:06.079117 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.079120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.079220 (+ 100us) server_negotiation.cc:407] Connection header received
0927 12:38:06.079361 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.079363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.079469 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.079562 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:38:06.079863 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.080416 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.081327 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.081564 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.081651 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.082003 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:38:06.082458 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:38:06.082560 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.082769 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.082853 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:38:06.083689 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:38:06.087327 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:38:06.095970 14952 tablet_service.cc:1467] Processing CreateTablet for tablet a61159b5f2fb40d7858461afbb37cc01 (DEFAULT_TABLE table=test-table [id=fc215b1baf3d44cabf453b5eb75717e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:06.095970 14813 tablet_service.cc:1467] Processing CreateTablet for tablet a61159b5f2fb40d7858461afbb37cc01 (DEFAULT_TABLE table=test-table [id=fc215b1baf3d44cabf453b5eb75717e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:06.096668 14813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61159b5f2fb40d7858461afbb37cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:06.096669 14952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61159b5f2fb40d7858461afbb37cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:06.099785 14771 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Bootstrap starting.
I20260927 12:38:06.099784 14910 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Bootstrap starting.
I20260927 12:38:06.100129 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.095155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.0.1:37084)
0927 12:38:06.095303 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.095320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:06.095420 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:06.096569 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:06.096573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:06.096597 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:06.096707 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.096715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.097714 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.097718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.098681 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.098694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.099553 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.099557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.099658 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.099777 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 12:38:06.099854 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 12:38:06.099951 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 12:38:06.100836 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.095568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.98.3:41389)
0927 12:38:06.095989 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.095999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.096020 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:38:06.096125 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.096131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.096411 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.096577 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:06.096796 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.097583 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.098826 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.099417 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.099522 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.100570 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.100624 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.100719 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":181,"threads_started":1}
I20260927 12:38:06.100914 14910 tablet_bootstrap.cc:654] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:06.100946 14771 tablet_bootstrap.cc:654] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:06.103428 14910 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: No bootstrap required, opened a new log
I20260927 12:38:06.103485 14771 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: No bootstrap required, opened a new log
I20260927 12:38:06.103549 14910 ts_tablet_manager.cc:1397] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:38:06.103574 14771 ts_tablet_manager.cc:1397] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:38:06.104048 14771 raft_consensus.cc:348] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.104053 14910 raft_consensus.cc:348] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.104211 14910 raft_consensus.cc:374] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:06.104211 14771 raft_consensus.cc:374] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:06.104260 14771 raft_consensus.cc:729] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731b811510b14b7781f67a07f98f00a0, State: Initialized, Role: FOLLOWER
I20260927 12:38:06.104260 14910 raft_consensus.cc:729] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6bd7c240fb34d43a4dc1aa7e65fa735, State: Initialized, Role: FOLLOWER
I20260927 12:38:06.104398 14910 consensus_queue.cc:260] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.104454 14771 consensus_queue.cc:260] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.104678 14910 ts_tablet_manager.cc:1428] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:38:06.104174 15088 tablet_service.cc:1467] Processing CreateTablet for tablet a61159b5f2fb40d7858461afbb37cc01 (DEFAULT_TABLE table=test-table [id=fc215b1baf3d44cabf453b5eb75717e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:06.105443 15088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a61159b5f2fb40d7858461afbb37cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:06.105614 14771 ts_tablet_manager.cc:1428] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 12:38:06.117394 15166 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Bootstrap starting.
I20260927 12:38:06.120718 15166 tablet_bootstrap.cc:654] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:06.121661 15166 log.cc:824] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:06.124037 15166 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: No bootstrap required, opened a new log
I20260927 12:38:06.124326 15166 ts_tablet_manager.cc:1397] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20260927 12:38:06.127962 15166 raft_consensus.cc:348] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.128302 15166 raft_consensus.cc:374] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:06.128345 15166 raft_consensus.cc:729] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeea41cea85b454797ef2a082341b3c0, State: Initialized, Role: FOLLOWER
I20260927 12:38:06.128803 15166 consensus_queue.cc:260] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.129535 15154 heartbeater.cc:503] Master 127.0.98.62:34781 was elected leader, sending a full tablet report...
I20260927 12:38:06.129922 15166 ts_tablet_manager.cc:1428] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260927 12:38:06.301465 15155 tablet.cc:2369] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:38:06.311257 14885 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:06.311487 14885 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.312668 14885 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
W20260927 12:38:06.313408 15020 tablet.cc:2369] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:38:06.316733 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.313067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53241 (local address 127.0.98.3:41389)
0927 12:38:06.313220 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.313224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.313385 (+ 161us) server_negotiation.cc:407] Connection header received
0927 12:38:06.313676 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.313680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.313734 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.313806 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:06.314074 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.314576 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.315699 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.316160 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.316265 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.316341 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.316518 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.316597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:38:06.316771 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.313085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38061 (local address 127.0.98.2:38009)
0927 12:38:06.313219 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.313223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.313486 (+ 263us) server_negotiation.cc:407] Connection header received
0927 12:38:06.313675 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.313679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.313733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.313790 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:06.314133 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.314644 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.315747 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.316252 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.316346 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.316491 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.316546 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.316635 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:38:06.316831 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.313043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38009 (local address 127.0.98.1:38061)
0927 12:38:06.313389 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.313403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:06.313515 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:06.313931 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:06.313933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:06.313950 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:06.314021 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.314025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.314752 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.314755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.315636 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.315647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.316361 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.316364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.316448 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.316468 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:38:06.316532 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:38:06.316615 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":11,"thread_start_us":86,"threads_started":1}
I20260927 12:38:06.317056 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.312919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.98.1:53241)
0927 12:38:06.313275 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.313292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:06.313444 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:06.313830 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:06.313833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:06.313852 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:06.313968 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.313975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.314690 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.314694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.315582 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:06.315595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.316283 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.316287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:06.316377 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.316420 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 12:38:06.316504 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 12:38:06.316588 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260927 12:38:06.317497 14972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61159b5f2fb40d7858461afbb37cc01" candidate_uuid: "731b811510b14b7781f67a07f98f00a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" is_pre_election: true
I20260927 12:38:06.317606 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61159b5f2fb40d7858461afbb37cc01" candidate_uuid: "731b811510b14b7781f67a07f98f00a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeea41cea85b454797ef2a082341b3c0" is_pre_election: true
I20260927 12:38:06.317797 14972 raft_consensus.cc:2454] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731b811510b14b7781f67a07f98f00a0 in term 0.
I20260927 12:38:06.317896 15108 raft_consensus.cc:2454] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731b811510b14b7781f67a07f98f00a0 in term 0.
I20260927 12:38:06.318254 14766 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0, b6bd7c240fb34d43a4dc1aa7e65fa735; no voters:
I20260927 12:38:06.318518 14885 raft_consensus.cc:2781] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:38:06.318600 14885 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:38:06.318642 14885 raft_consensus.cc:3037] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:06.319602 14885 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.319921 14885 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 1 election: Requested vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
I20260927 12:38:06.320353 14972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61159b5f2fb40d7858461afbb37cc01" candidate_uuid: "731b811510b14b7781f67a07f98f00a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735"
I20260927 12:38:06.320479 14972 raft_consensus.cc:3037] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:06.321483 14972 raft_consensus.cc:2454] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731b811510b14b7781f67a07f98f00a0 in term 1.
I20260927 12:38:06.321738 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a61159b5f2fb40d7858461afbb37cc01" candidate_uuid: "731b811510b14b7781f67a07f98f00a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeea41cea85b454797ef2a082341b3c0"
I20260927 12:38:06.321790 14766 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0, b6bd7c240fb34d43a4dc1aa7e65fa735; no voters:
I20260927 12:38:06.321870 15108 raft_consensus.cc:3037] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:06.321959 14885 raft_consensus.cc:2781] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:06.322151 14885 raft_consensus.cc:686] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 LEADER]: Becoming Leader. State: Replica: 731b811510b14b7781f67a07f98f00a0, State: Running, Role: LEADER
I20260927 12:38:06.322427 14885 consensus_queue.cc:237] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.323022 15108 raft_consensus.cc:2454] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731b811510b14b7781f67a07f98f00a0 in term 1.
I20260927 12:38:06.325374 14691 catalog_manager.cc:5495] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731b811510b14b7781f67a07f98f00a0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "731b811510b14b7781f67a07f98f00a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } health_report { overall_health: UNKNOWN } } }
I20260927 12:38:06.347179 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.343580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.98.1:35817)
0927 12:38:06.343878 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:06.343886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:06.343899 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:38:06.343973 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:06.343976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:06.344051 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:06.344189 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:38:06.344402 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.345026 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.345798 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:06.346027 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:06.346114 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:06.346362 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:38:06.346835 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:38:06.346936 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:06.346987 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 12:38:06.347045 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":125,"threads_started":1}
I20260927 12:38:06.351033 15108 raft_consensus.cc:1261] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Refusing update from remote peer 731b811510b14b7781f67a07f98f00a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:06.351042 14972 raft_consensus.cc:1261] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Refusing update from remote peer 731b811510b14b7781f67a07f98f00a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:06.351740 15171 consensus_queue.cc:1035] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:06.351918 14885 consensus_queue.cc:1035] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:38:06.352106 14880 tablet.cc:2369] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:38:06.357460 15176 mvcc.cc:205] Tried to move back new op lower bound from 7333939963287752704 to 7333939963179573248. Current Snapshot: MvccSnapshot[applied={T|T < 7333939963287752704}]
I20260927 12:38:06.367237 14691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:06.367408 14691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:06.370467 14691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Sending DeleteTablet for 3 replicas of tablet a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:06.371351 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet a61159b5f2fb40d7858461afbb37cc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38898
I20260927 12:38:06.371428 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet a61159b5f2fb40d7858461afbb37cc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34396
I20260927 12:38:06.371441 15088 tablet_service.cc:1514] Processing DeleteTablet for tablet a61159b5f2fb40d7858461afbb37cc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37084
I20260927 12:38:06.371999 15186 tablet_replica.cc:330] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: stopping tablet replica
I20260927 12:38:06.372128 15187 tablet_replica.cc:330] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: stopping tablet replica
I20260927 12:38:06.372201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14729
I20260927 12:38:06.372234 15186 raft_consensus.cc:2229] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:38:06.372475 15187 raft_consensus.cc:2229] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:06.372807 15185 tablet_replica.cc:330] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: stopping tablet replica
I20260927 12:38:06.372941 15185 raft_consensus.cc:2229] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:06.372961 15187 raft_consensus.cc:2258] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:06.373092 15185 raft_consensus.cc:2258] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:06.374051 15185 ts_tablet_manager.cc:1905] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:06.374524 15187 ts_tablet_manager.cc:1905] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:06.376855 15185 ts_tablet_manager.cc:1918] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:06.376946 15185 log.cc:1196] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:06.377177 15185 ts_tablet_manager.cc:1939] T a61159b5f2fb40d7858461afbb37cc01 P b6bd7c240fb34d43a4dc1aa7e65fa735: Deleting consensus metadata
I20260927 12:38:06.378014 14676 catalog_manager.cc:4841] TS b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): tablet a61159b5f2fb40d7858461afbb37cc01 (table test-table [id=fc215b1baf3d44cabf453b5eb75717e5]) successfully deleted
I20260927 12:38:06.378839 15187 ts_tablet_manager.cc:1918] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:06.378959 15187 log.cc:1196] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:06.379288 15187 ts_tablet_manager.cc:1939] T a61159b5f2fb40d7858461afbb37cc01 P eeea41cea85b454797ef2a082341b3c0: Deleting consensus metadata
W20260927 12:38:06.380232 14676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35817 (error 108)
I20260927 12:38:06.380960 14675 catalog_manager.cc:4841] TS eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): tablet a61159b5f2fb40d7858461afbb37cc01 (table test-table [id=fc215b1baf3d44cabf453b5eb75717e5]) successfully deleted
I20260927 12:38:06.381072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14884
I20260927 12:38:06.381350 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.381052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35817 (local address 127.0.0.1:38922)
0927 12:38:06.381181 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.381274 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 12:38:06.382144 14676 catalog_manager.cc:4584] TS 731b811510b14b7781f67a07f98f00a0 (127.0.98.1:35817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a61159b5f2fb40d7858461afbb37cc01: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35817: connect: Connection refused (error 111)
I20260927 12:38:06.386452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15022
I20260927 12:38:06.391834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 14658
I20260927 12:38:06.398494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34781
--webserver_interface=127.0.98.62
--webserver_port=40545
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:06.514743 15189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:06.515096 15189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:06.515164 15189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:06.519662 15189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:06.519760 15189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:06.519780 15189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:06.519796 15189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:06.519810 15189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:06.526269 15189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:06.527427 15189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:06.528573 15189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:06.538414 15195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:06.538414 15194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:06.538414 15197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:06.539165 15189 server_base.cc:1034] running on GCE node
I20260927 12:38:06.539814 15189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:06.541455 15189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:06.542783 15189 hybrid_clock.cc:648] HybridClock initialized: now 1790512686542745 us; error 54 us; skew 500 ppm
Sep 27 12:38:06 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:06.546268 15189 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:38:06.547446 15189 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260927 12:38:06.548054 15189 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:06.548182 15189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:06.552470 15189 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:06.554816 15204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:06.556300 15189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:06.556434 15189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "26fb8700a7a1418488ccb98d8877fd93"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "384ca87ca275c2a77e9e9692c7eaac06"
server_key_iv: "7db49652a2b04f30fd9ffd714013b5af"
server_key_version: "encryptionkey@0"
I20260927 12:38:06.557031 15189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:06.576452 15189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:06.580214 15189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:06.580425 15189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:06.587788 15189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34781
I20260927 12:38:06.588248 15256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34781 every 8 connection(s)
I20260927 12:38:06.589021 15189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:38:06.593552 15257 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:38:06.595322 15257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap starting.
I20260927 12:38:06.595870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15189
I20260927 12:38:06.596342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35817
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34781
--webserver_port=33111
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:06.614717 15257 log.cc:824] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:06.626240 15257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:06.626631 15257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93: Bootstrap complete.
I20260927 12:38:06.631117 15257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:06.631579 15257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fb8700a7a1418488ccb98d8877fd93, State: Initialized, Role: FOLLOWER
I20260927 12:38:06.632180 15257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:06.632313 15257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:38:06.632362 15257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:38:06.632459 15257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:38:06.634497 15257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:06.634864 15257 leader_election.cc:302] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26fb8700a7a1418488ccb98d8877fd93; no voters:
I20260927 12:38:06.635174 15257 leader_election.cc:288] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 12:38:06.635341 15261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:38:06.635576 15261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [term 3 LEADER]: Becoming Leader. State: Replica: 26fb8700a7a1418488ccb98d8877fd93, State: Running, Role: LEADER
I20260927 12:38:06.635897 15261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } }
I20260927 12:38:06.636436 15257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:06.637355 15262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "26fb8700a7a1418488ccb98d8877fd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } } }
I20260927 12:38:06.637482 15262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:06.637746 15263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26fb8700a7a1418488ccb98d8877fd93. Latest consensus state: current_term: 3 leader_uuid: "26fb8700a7a1418488ccb98d8877fd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fb8700a7a1418488ccb98d8877fd93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34781 } } }
I20260927 12:38:06.637821 15263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fb8700a7a1418488ccb98d8877fd93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:06.638345 15270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:06.642944 15270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:06.643535 15270 catalog_manager.cc:1259] Loaded cluster ID: cd641c3f981846d799e96873feb5e965
I20260927 12:38:06.643592 15270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 12:38:06.644477 15270 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7a159fa9ad google::LogMessage::Fail() at ??:0
@ 0x7f7a159fd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7a159fa4de google::LogMessage::Flush() at ??:0
@ 0x7f7a159fba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7a20339fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7a2033ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7a20337579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7a2037b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7a17c4c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7a17c4cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7a17c4e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7a17c3d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7a192816db start_thread at ??:0
@ 0x7f7a144f861f clone at ??:0
W20260927 12:38:06.731230 15259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:06.731523 15259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:06.731595 15259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:06.736084 15259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:06.736224 15259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:06.736311 15259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:38:06.742002 15259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33111
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:06.743155 15259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:06.744264 15259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:06.754649 15298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:06.754662 15301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:06.754703 15299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:06.754940 15259 server_base.cc:1034] running on GCE node
I20260927 12:38:06.756139 15259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:06.756933 15259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:06.758148 15259 hybrid_clock.cc:648] HybridClock initialized: now 1790512686758112 us; error 62 us; skew 500 ppm
Sep 27 12:38:06 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:06.761440 15259 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:38:06.762759 15259 webserver.cc:460] Webserver started at http://127.0.98.1:33111/ using document root <none> and password file <none>
I20260927 12:38:06.763491 15259 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:06.763602 15259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:06.770196 15259 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:38:06.773252 15308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:06.774967 15259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:38:06.775143 15259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "731b811510b14b7781f67a07f98f00a0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "cb0b971baecf768895b42aca060e3c92"
server_key_iv: "ce280ae59a3db4f2fed8808120c3f6a5"
server_key_version: "encryptionkey@0"
I20260927 12:38:06.775980 15259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:06.793509 15259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:06.797876 15259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:06.798112 15259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:06.798949 15259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:06.800527 15315 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:38:06.804857 15259 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:38:06.804962 15259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 12:38:06.805037 15259 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:38:06.807632 15259 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:38:06.807727 15259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260927 12:38:06.808182 15315 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Bootstrap starting.
I20260927 12:38:06.819057 15259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35817
I20260927 12:38:06.819269 15422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35817 every 8 connection(s)
I20260927 12:38:06.820538 15259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:38:06.824359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15259
I20260927 12:38:06.824841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38009
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34781
--webserver_port=37135
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:06.831722 15315 log.cc:824] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:06.841601 15315 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:06.842062 15315 tablet_bootstrap.cc:492] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Bootstrap complete.
I20260927 12:38:06.843400 15315 ts_tablet_manager.cc:1397] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.004s
I20260927 12:38:06.846971 15315 raft_consensus.cc:348] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.847563 15315 raft_consensus.cc:729] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731b811510b14b7781f67a07f98f00a0, State: Initialized, Role: FOLLOWER
I20260927 12:38:06.848202 15315 consensus_queue.cc:260] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:06.849438 15315 ts_tablet_manager.cc:1428] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 12:38:06.902693 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:38:06.823028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:43175)
0927 12:38:06.823645 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.823715 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:06.824765 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:06.901875 (+ 77110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":182,"threads_started":1}
W20260927 12:38:06.903227 15423 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34781 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34781: Client connection negotiation failed: client connection to 127.0.98.62:34781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:38:06.904018 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.903734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:54395)
0927 12:38:06.903887 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.903957 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 12:38:06.904687 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:06.904477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:56885)
0927 12:38:06.904581 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:06.904625 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:38:06.904848 15423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:38:06.949863 15426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:06.950167 15426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:06.950229 15426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:06.954582 15426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:06.954660 15426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:06.954756 15426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:38:06.960465 15426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37135
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:06.961606 15426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:06.962672 15426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:06.972642 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:06.972687 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:06.972800 15426 server_base.cc:1034] running on GCE node
W20260927 12:38:06.972645 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:06.973361 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:06.974059 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:06.975235 15426 hybrid_clock.cc:648] HybridClock initialized: now 1790512686975211 us; error 48 us; skew 500 ppm
Sep 27 12:38:06 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:06.977981 15426 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:38:06.978873 15426 webserver.cc:460] Webserver started at http://127.0.98.2:37135/ using document root <none> and password file <none>
I20260927 12:38:06.979410 15426 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:06.979480 15426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:06.983601 15426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:06.986048 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:06.987352 15426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:06.987483 15426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "3f633fd0d7f6ee1b9268491c9386bdf1"
server_key_iv: "89a3dd5541e1893c9dc771b1e9887839"
server_key_version: "encryptionkey@0"
I20260927 12:38:06.988243 15426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:07.000694 15426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:07.004411 15426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:07.004621 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:07.005316 15426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:07.006178 15426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:07.006225 15426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:07.006265 15426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:07.006285 15426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:07.015589 15426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38009
I20260927 12:38:07.015784 15556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38009 every 8 connection(s)
I20260927 12:38:07.016810 15426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:38:07.021045 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.019142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:35403)
0927 12:38:07.019704 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.020260 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":123,"threads_started":1}
W20260927 12:38:07.021516 15557 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34781 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34781: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
I20260927 12:38:07.022171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15426
I20260927 12:38:07.022205 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.021929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:56067)
0927 12:38:07.022076 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.022130 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 12:38:07.022588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41389
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34781
--webserver_port=42899
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:07.022908 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.022641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:36067)
0927 12:38:07.022787 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.022842 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 12:38:07.023095 15557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:38:07.140647 15560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:07.140944 15560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:07.140990 15560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:07.145169 15560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:07.145251 15560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:07.145313 15560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:38:07.150918 15560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42899
--tserver_master_addrs=127.0.98.62:34781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:07.152037 15560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:07.153077 15560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:07.164033 15565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:07.164263 15560 server_base.cc:1034] running on GCE node
W20260927 12:38:07.163952 15568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:07.163952 15566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:07.165037 15560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:07.165787 15560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:07.166981 15560 hybrid_clock.cc:648] HybridClock initialized: now 1790512687166977 us; error 39 us; skew 500 ppm
Sep 27 12:38:07 dist-test-slave-tzgx krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:07.170280 15560 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:38:07.171394 15560 webserver.cc:460] Webserver started at http://127.0.98.3:42899/ using document root <none> and password file <none>
I20260927 12:38:07.171998 15560 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:07.172073 15560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:07.176703 15560 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:07.179245 15575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:07.180861 15560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:07.181010 15560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "eeea41cea85b454797ef2a082341b3c0"
format_stamp: "Formatted at 2026-09-27 12:38:04 on dist-test-slave-tzgx"
server_key: "15cff7a97f3b10ddaccdf61812e6b47e"
server_key_iv: "cf5d71c3ef6241eba74798ccd74620e8"
server_key_version: "encryptionkey@0"
I20260927 12:38:07.181648 15560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:07.205082 15560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:07.208773 15560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:07.208992 15560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:07.209656 15560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:07.210534 15560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:07.210582 15560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:07.210646 15560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:07.210675 15560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:07.220073 15560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41389
I20260927 12:38:07.220124 15688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41389 every 8 connection(s)
I20260927 12:38:07.221136 15560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:38:07.225219 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.223325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:48191)
0927 12:38:07.223845 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.223967 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":111,"threads_started":1}
W20260927 12:38:07.225796 15689 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34781 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34781: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
I20260927 12:38:07.226483 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.226221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:45933)
0927 12:38:07.226369 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.226423 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 12:38:07.227146 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.226941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:59581)
0927 12:38:07.227049 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.227092 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:38:07.227308 15689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:38:07.230057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15560
I20260927 12:38:07.905953 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:07.905473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:51051)
0927 12:38:07.905780 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:07.905845 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260927 12:38:08.024329 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.023801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:41525)
0927 12:38:08.024142 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:08.024214 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260927 12:38:08.228452 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.227962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:33037)
0927 12:38:08.228265 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:08.228328 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260927 12:38:08.403887 15696 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:08.404124 15696 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:08.405272 15696 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
Sep 27 12:38:08 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 12:38:08 dist-test-slave-tzgx krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 12:38:08.420439 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.405761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37857 (local address 127.0.98.3:41389)
0927 12:38:08.406205 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:08.406228 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:08.406268 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:38:08.406928 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:08.406933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:08.407288 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:08.407553 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:08.408941 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.410538 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.412680 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.413297 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.413514 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:08.415891 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:08.415897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:08.415908 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:08.415938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:08.417798 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:08.418362 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:08.418367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:08.418371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:08.418482 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:08.418850 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:08.418853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:08.418855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:08.419286 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:08.419407 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:08.419958 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 12:38:08.420259 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":107,"threads_started":1}
I20260927 12:38:08.420430 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.405519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38009 (local address 127.0.98.1:33407)
0927 12:38:08.405702 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:08.405718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:08.405824 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:08.407679 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:08.407685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:08.408200 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:08.408751 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:08.408769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.410743 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.410746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:08.411798 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:08.411812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.412756 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.412758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:08.412897 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:08.413636 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:08.413662 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:08.415402 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:08.417775 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:08.417782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:08.417798 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:08.418139 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:08.418612 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:08.418615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:08.418616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:08.418763 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:08.419407 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:08.419414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:08.419655 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 12:38:08.419822 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:38:08.420214 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":55,"spinlock_wait_cycles":4608}
I20260927 12:38:08.420507 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.405757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33407 (local address 127.0.98.2:38009)
0927 12:38:08.406239 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:08.406257 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:08.406288 (+ 31us) server_negotiation.cc:407] Connection header received
0927 12:38:08.406926 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:08.406932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:08.407288 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:08.407567 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:08.408956 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.410538 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.411951 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.412585 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.412794 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:08.415699 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:08.415707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:08.415725 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:08.415763 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:08.417641 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:08.418273 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:08.418281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:08.418287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:08.418505 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:08.418859 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:08.418863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:08.418865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:08.419287 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:08.419417 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:08.419966 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 12:38:08.420282 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":111,"threads_started":1}
I20260927 12:38:08.420487 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.405519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41389 (local address 127.0.98.1:37857)
0927 12:38:08.406036 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:08.406048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:08.406136 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:08.407940 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:08.407943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:08.408205 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:08.408751 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:08.408769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.411962 (+ 3193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.411964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:08.412547 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:08.412557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:08.413462 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:08.413465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:08.413608 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:08.414010 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:08.414025 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:08.415708 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:08.417941 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:08.417944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:08.417946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:08.418246 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:08.418580 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:08.418584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:08.418586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:08.418754 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:08.419408 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:08.419414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:08.419655 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 12:38:08.419823 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 12:38:08.420294 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":368,"spinlock_wait_cycles":6272,"thread_start_us":74,"threads_started":1}
W20260927 12:38:08.423560 15310 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
W20260927 12:38:08.424136 15309 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:08.424217 15309 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0; no voters: b6bd7c240fb34d43a4dc1aa7e65fa735, eeea41cea85b454797ef2a082341b3c0
I20260927 12:38:08.424450 15696 raft_consensus.cc:2726] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:38:08.907065 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:08.906755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:39897)
0927 12:38:08.906914 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:08.906980 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 12:38:09.025553 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:09.025100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:47433)
0927 12:38:09.025363 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:09.025421 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260927 12:38:09.229830 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:09.229311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:47979)
0927 12:38:09.229611 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:09.229676 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260927 12:38:09.908613 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:09.907966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:51661)
0927 12:38:09.908420 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:09.908499 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":83,"threads_started":1}
I20260927 12:38:10.026957 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:10.026443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:54709)
0927 12:38:10.026777 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:10.026848 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":133,"threads_started":1}
I20260927 12:38:10.153033 15704 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:10.153165 15704 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:10.153586 15704 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
W20260927 12:38:10.154408 15309 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
W20260927 12:38:10.154510 15310 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:10.154613 15310 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0; no voters: b6bd7c240fb34d43a4dc1aa7e65fa735, eeea41cea85b454797ef2a082341b3c0
I20260927 12:38:10.154841 15704 raft_consensus.cc:2726] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:38:10.231235 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:10.230699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:41009)
0927 12:38:10.231024 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:10.231104 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260927 12:38:10.910045 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:10.909585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:35387)
0927 12:38:10.909870 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:10.909937 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260927 12:38:11.028671 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:11.027997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:54151)
0927 12:38:11.028461 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:11.028528 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":148,"threads_started":1}
I20260927 12:38:11.232726 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:11.232232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:51131)
0927 12:38:11.232555 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:11.232620 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260927 12:38:11.911401 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:11.910933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:45543)
0927 12:38:11.911248 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:11.911307 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
I20260927 12:38:12.030081 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:12.029645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:60687)
0927 12:38:12.029915 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:12.029985 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260927 12:38:12.234275 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:12.233782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:37103)
0927 12:38:12.234109 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:12.234175 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":88,"threads_started":1}
I20260927 12:38:12.271986 15712 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:12.272183 15712 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:12.272612 15712 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
W20260927 12:38:12.273384 15309 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
W20260927 12:38:12.273485 15310 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:12.273568 15310 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0; no voters: b6bd7c240fb34d43a4dc1aa7e65fa735, eeea41cea85b454797ef2a082341b3c0
I20260927 12:38:12.273703 15712 raft_consensus.cc:2726] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:38:12.912787 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:12.912329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:37511)
0927 12:38:12.912622 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:12.912688 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260927 12:38:13.031471 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:13.030910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:39985)
0927 12:38:13.031217 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:13.031280 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260927 12:38:13.235620 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:13.235111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:43369)
0927 12:38:13.235444 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:13.235519 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260927 12:38:13.914180 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:13.913683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:43799)
0927 12:38:13.913984 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:13.914054 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20260927 12:38:14.032821 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:14.032371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:43859)
0927 12:38:14.032654 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:14.032723 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260927 12:38:14.236984 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:14.236483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:49227)
0927 12:38:14.236821 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:14.236885 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
I20260927 12:38:14.393464 15719 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:14.393603 15719 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:14.394025 15719 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
W20260927 12:38:14.394768 15310 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
W20260927 12:38:14.394883 15309 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:14.394973 15309 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0; no voters: b6bd7c240fb34d43a4dc1aa7e65fa735, eeea41cea85b454797ef2a082341b3c0
I20260927 12:38:14.395128 15719 raft_consensus.cc:2726] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:38:14.915441 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:14.915009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:56377)
0927 12:38:14.915255 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:14.915313 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260927 12:38:15.034185 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:15.033720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:34409)
0927 12:38:15.034012 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:15.034079 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260927 12:38:15.238252 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:15.237793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:57951)
0927 12:38:15.238073 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:15.238149 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260927 12:38:15.916772 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:15.916300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:51945)
0927 12:38:15.916614 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:15.916679 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":80,"threads_started":1}
I20260927 12:38:16.035471 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:16.035020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:54551)
0927 12:38:16.035312 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:16.035377 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260927 12:38:16.239590 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:16.239139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:36177)
0927 12:38:16.239432 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:16.239494 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260927 12:38:16.918187 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:16.917670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:51357)
0927 12:38:16.917979 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:16.918065 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260927 12:38:17.037074 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:17.036617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:51237)
0927 12:38:17.036902 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:17.036976 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260927 12:38:17.241024 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:17.240582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:36997)
0927 12:38:17.240859 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:17.240926 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260927 12:38:17.919592 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:17.919067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:33531)
0927 12:38:17.919355 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:17.919423 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260927 12:38:17.941864 15730 raft_consensus.cc:482] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:17.941991 15730 raft_consensus.cc:504] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeea41cea85b454797ef2a082341b3c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41389 } } peers { permanent_uuid: "731b811510b14b7781f67a07f98f00a0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35817 } } peers { permanent_uuid: "b6bd7c240fb34d43a4dc1aa7e65fa735" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38009 } }
I20260927 12:38:17.942425 15730 leader_election.cc:288] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389), b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009)
W20260927 12:38:17.943257 15309 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeea41cea85b454797ef2a082341b3c0 (127.0.98.3:41389): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
W20260927 12:38:17.943373 15310 leader_election.cc:341] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6bd7c240fb34d43a4dc1aa7e65fa735 (127.0.98.2:38009): Not found: Tablet not found: a61159b5f2fb40d7858461afbb37cc01
I20260927 12:38:17.943439 15310 leader_election.cc:302] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 731b811510b14b7781f67a07f98f00a0; no voters: b6bd7c240fb34d43a4dc1aa7e65fa735, eeea41cea85b454797ef2a082341b3c0
I20260927 12:38:17.943588 15730 raft_consensus.cc:2726] T a61159b5f2fb40d7858461afbb37cc01 P 731b811510b14b7781f67a07f98f00a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:38:18.038389 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:18.037869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:49039)
0927 12:38:18.038204 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:18.038275 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":78,"threads_started":1}
I20260927 12:38:18.242441 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:18.241980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:51011)
0927 12:38:18.242279 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:18.242342 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260927 12:38:18.921015 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:18.920564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:42883)
0927 12:38:18.920842 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:18.920905 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260927 12:38:19.039726 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:19.039207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:47935)
0927 12:38:19.039556 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:19.039625 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260927 12:38:19.243801 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:19.243318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:47535)
0927 12:38:19.243633 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:19.243703 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":78,"threads_started":1}
I20260927 12:38:19.922366 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:19.921901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:56783)
0927 12:38:19.922182 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:19.922249 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
I20260927 12:38:20.041249 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:20.040732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:37699)
0927 12:38:20.041084 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:20.041150 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":77,"threads_started":1}
I20260927 12:38:20.245052 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:20.244615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:44503)
0927 12:38:20.244876 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:20.244935 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260927 12:38:20.923633 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:20.923177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:60751)
0927 12:38:20.923469 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:20.923535 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260927 12:38:21.042529 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:21.042087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:45153)
0927 12:38:21.042341 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:21.042401 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260927 12:38:21.246338 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:21.245896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:52439)
0927 12:38:21.246175 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:21.246241 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260927 12:38:21.925146 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:21.924701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.1:34435)
0927 12:38:21.924972 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:21.925031 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260927 12:38:22.044070 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:22.043595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.2:52333)
0927 12:38:22.043879 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:22.043946 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260927 12:38:22.241062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15259
I20260927 12:38:22.246582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15426
I20260927 12:38:22.248440 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:22.247893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34781 (local address 127.0.98.3:48641)
0927 12:38:22.248256 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:22.248326 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260927 12:38:22.252218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15560
2026-09-27T12:38:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19428 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:38:22 dist-test-slave-tzgx krb5kdc[15747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:38:22 dist-test-slave-tzgx krb5kdc[15747](info): set up 2 sockets
Sep 27 12:38:22 dist-test-slave-tzgx krb5kdc[15747](info): commencing operation
krb5kdc: starting...
W20260927 12:38:23.244280 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:38:23 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:38:23.371893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:23.492770 15760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:23.493085 15760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:23.493132 15760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:23.497434 15760 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:38:23.497535 15760 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:38:23.497560 15760 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:38:23.497574 15760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:23.497593 15760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:23.497612 15760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:23.497641 15760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:23.497671 15760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:23.503419 15760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:23.504601 15760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:23.505622 15760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:23.514757 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:23.514789 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:23.514734 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:23.515142 15760 server_base.cc:1034] running on GCE node
I20260927 12:38:23.515692 15760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:23.516875 15760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:23.518067 15760 hybrid_clock.cc:648] HybridClock initialized: now 1790512703518039 us; error 41 us; skew 500 ppm
Sep 27 12:38:23 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:23.521028 15760 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:38:23.522042 15760 webserver.cc:460] Webserver started at http://127.0.98.62:37895/ using document root <none> and password file <none>
I20260927 12:38:23.522607 15760 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:23.522679 15760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:23.522887 15760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:23.525094 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "7b1fa084526d4d4585c562f7b5c938c6"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "1aab24d57442702218edeb97dcb4c1d4"
server_key_iv: "b3aaa840a0fbc9abdd3d0f8ce1c455dd"
server_key_version: "encryptionkey@0"
I20260927 12:38:23.525694 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b1fa084526d4d4585c562f7b5c938c6"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "1aab24d57442702218edeb97dcb4c1d4"
server_key_iv: "b3aaa840a0fbc9abdd3d0f8ce1c455dd"
server_key_version: "encryptionkey@0"
I20260927 12:38:23.529959 15760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:38:23.533425 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:23.535455 15760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:38:23.535604 15760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "7b1fa084526d4d4585c562f7b5c938c6"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "1aab24d57442702218edeb97dcb4c1d4"
server_key_iv: "b3aaa840a0fbc9abdd3d0f8ce1c455dd"
server_key_version: "encryptionkey@0"
I20260927 12:38:23.535725 15760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:23.556061 15760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:23.560348 15760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:23.560532 15760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:23.567852 15760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34847
I20260927 12:38:23.568221 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34847 every 8 connection(s)
I20260927 12:38:23.568943 15760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:38:23.571947 15828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:23.577878 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap starting.
I20260927 12:38:23.579016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15760
I20260927 12:38:23.579162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:38:23.579466 392 external_mini_cluster.cc:1428] Setting key 30810eff5e685a0832c7c1bdf69eebfe
I20260927 12:38:23.581084 15828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:23.582027 15828 log.cc:824] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:23.584621 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: No bootstrap required, opened a new log
Sep 27 12:38:23 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:23.588910 15828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:23.589174 15828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:23.589242 15828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1fa084526d4d4585c562f7b5c938c6, State: Initialized, Role: FOLLOWER
I20260927 12:38:23.589826 15828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:23.590016 15828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:38:23.590103 15828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:38:23.590212 15828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:23.591497 15828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:23.591933 15828 leader_election.cc:302] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b1fa084526d4d4585c562f7b5c938c6; no voters:
I20260927 12:38:23.592486 15828 leader_election.cc:288] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:38:23.592638 15833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:23.592839 15833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 LEADER]: Becoming Leader. State: Replica: 7b1fa084526d4d4585c562f7b5c938c6, State: Running, Role: LEADER
I20260927 12:38:23.593245 15833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:23.593921 15828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:23.594666 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b1fa084526d4d4585c562f7b5c938c6. Latest consensus state: current_term: 1 leader_uuid: "7b1fa084526d4d4585c562f7b5c938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } }
I20260927 12:38:23.594800 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:23.595312 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b1fa084526d4d4585c562f7b5c938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } }
I20260927 12:38:23.595619 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:23.595695 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:23.580704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.0.98.62:34847)
0927 12:38:23.581183 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:23.581203 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:23.581248 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:38:23.582161 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:23.582186 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:23.582615 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:23.582927 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:23.583689 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:23.585173 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:23.586154 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:23.586903 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:23.587090 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:23.589864 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:23.589871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:23.589889 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:23.589932 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:23.592000 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:23.592665 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:23.592671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:23.592676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:23.592857 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:23.593177 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:23.593180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:23.593181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:23.593656 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:23.593866 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:23.594173 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 12:38:23.594682 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":132,"threads_started":1}
I20260927 12:38:23.595789 15842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:23.599340 15842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:23.605934 15842 catalog_manager.cc:1347] Generated new cluster ID: 17d5a043628c444595efa6ed071c30a4
I20260927 12:38:23.606032 15842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:38:23.614776 15842 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:38:23.615823 15842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:38:23.625761 15842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Generated new TSK 0
I20260927 12:38:23.626959 15842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:38:23.671145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34847
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:23.790920 15854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:23.791213 15854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:23.791260 15854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:23.795739 15854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:23.795820 15854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:23.795888 15854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:38:23.802387 15854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:23.804158 15854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:23.805666 15854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:23.817096 15860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:23.817168 15862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:23.817096 15859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:23.817530 15854 server_base.cc:1034] running on GCE node
I20260927 12:38:23.818065 15854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:23.819015 15854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:23.820191 15854 hybrid_clock.cc:648] HybridClock initialized: now 1790512703820102 us; error 105 us; skew 500 ppm
Sep 27 12:38:23 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:23.823248 15854 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:38:23.824364 15854 webserver.cc:460] Webserver started at http://127.0.98.1:45309/ using document root <none> and password file <none>
I20260927 12:38:23.824939 15854 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:23.825021 15854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:23.825233 15854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:23.827430 15854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed8396af3ee64b93b9230c8be15e5b5c"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "066e0cbbbdbc9cc917a86f2dc85a5dfa"
server_key_iv: "ac017d09319f61453c6219256cacc165"
server_key_version: "encryptionkey@0"
I20260927 12:38:23.828265 15854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed8396af3ee64b93b9230c8be15e5b5c"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "066e0cbbbdbc9cc917a86f2dc85a5dfa"
server_key_iv: "ac017d09319f61453c6219256cacc165"
server_key_version: "encryptionkey@0"
I20260927 12:38:23.832398 15854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:38:23.835008 15869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:23.836447 15854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:23.836607 15854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "ed8396af3ee64b93b9230c8be15e5b5c"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "066e0cbbbdbc9cc917a86f2dc85a5dfa"
server_key_iv: "ac017d09319f61453c6219256cacc165"
server_key_version: "encryptionkey@0"
I20260927 12:38:23.836728 15854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:23.854097 15854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:23.857678 15854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:23.857878 15854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:23.858485 15854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:23.859316 15854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:23.859362 15854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:23.859414 15854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:23.859443 15854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:23.868253 15854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43577
I20260927 12:38:23.868290 15982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43577 every 8 connection(s)
I20260927 12:38:23.869177 15854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 12:38:23 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:23.878871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15854
I20260927 12:38:23.878997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:38:23.879258 392 external_mini_cluster.cc:1428] Setting key 2c4426919796b6e33d824507e27077d0
I20260927 12:38:23.883296 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:23.871125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37075 (local address 127.0.98.62:34847)
0927 12:38:23.871271 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:23.871275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:23.872398 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 12:38:23.873302 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:23.873306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:23.873355 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:23.873432 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:23.874663 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:23.875226 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:23.876469 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:23.876709 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:23.876792 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:23.879510 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:23.879514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:23.879516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:23.879545 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:23.881203 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:23.881757 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:23.881761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:23.881763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:23.881844 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:23.882189 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:23.882193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:23.882195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:23.882387 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:23.882472 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:23.883005 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 12:38:23.883132 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:38:23.884202 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:23.871505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:37075)
0927 12:38:23.872248 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:23.872296 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:23.873093 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:23.873592 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:23.873600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:23.873967 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:23.874498 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:23.874512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:23.875367 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:23.875370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:23.876358 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:23.876368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:23.876867 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:23.876870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:23.877022 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:23.877588 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:23.877606 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:23.879258 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:23.881326 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:23.881333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:23.881352 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:23.881659 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:23.881930 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:23.881933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:23.881935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:23.882102 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:23.882488 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:23.882495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:23.882768 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 12:38:23.882950 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 12:38:23.883316 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":112,"threads_started":1}
I20260927 12:38:23.885610 15983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34847
I20260927 12:38:23.885851 15983 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:23.886360 15983 heartbeater.cc:511] Master 127.0.98.62:34847 requested a full tablet report, sending...
I20260927 12:38:23.888001 15791 ts_manager.cc:194] Registered new tserver with Master: ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:38:23.889628 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:38:23.909253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34847
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:24.029194 15988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:24.029500 15988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:24.029547 15988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:24.034035 15988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:24.034122 15988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:24.034188 15988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:38:24.041782 15988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:24.042965 15988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:24.044224 15988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:24.054967 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:24.055050 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:24.055111 15988 server_base.cc:1034] running on GCE node
W20260927 12:38:24.055397 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:24.055867 15988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:24.056599 15988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:24.057793 15988 hybrid_clock.cc:648] HybridClock initialized: now 1790512704057759 us; error 49 us; skew 500 ppm
Sep 27 12:38:24 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:24.060631 15988 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:38:24.061564 15988 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260927 12:38:24.062114 15988 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:24.062172 15988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:24.062337 15988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:24.064499 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "52290e478d3e427d953202f45c6d7840"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "de4f5f8410329b8ec53af0bcf93a7e29"
server_key_iv: "e1b762e24544c276d665a6fe3b81a699"
server_key_version: "encryptionkey@0"
I20260927 12:38:24.065042 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "52290e478d3e427d953202f45c6d7840"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "de4f5f8410329b8ec53af0bcf93a7e29"
server_key_iv: "e1b762e24544c276d665a6fe3b81a699"
server_key_version: "encryptionkey@0"
I20260927 12:38:24.068951 15988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:38:24.071545 16003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:24.072994 15988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:38:24.073115 15988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "52290e478d3e427d953202f45c6d7840"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "de4f5f8410329b8ec53af0bcf93a7e29"
server_key_iv: "e1b762e24544c276d665a6fe3b81a699"
server_key_version: "encryptionkey@0"
I20260927 12:38:24.073230 15988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:24.088210 15988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:24.091203 15988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:24.091388 15988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:24.091998 15988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:24.092912 15988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:24.092959 15988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:24.093000 15988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:24.093014 15988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:24.101953 15988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41001
I20260927 12:38:24.102020 16116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41001 every 8 connection(s)
I20260927 12:38:24.102885 15988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:38:24.106272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 15988
I20260927 12:38:24.106389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:38:24.106683 392 external_mini_cluster.cc:1428] Setting key f46575ae3a18b1a4ef10da96d3105403
Sep 27 12:38:24 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:38:24.118153 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.104844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49513 (local address 127.0.98.62:34847)
0927 12:38:24.104990 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.104995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.106252 (+ 1257us) server_negotiation.cc:407] Connection header received
0927 12:38:24.107367 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.107371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.107435 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.107508 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:24.108971 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.109526 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.111245 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.111593 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.111707 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.114427 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:24.114431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:24.114442 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:24.114473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:24.115922 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:24.116517 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:24.116520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:24.116521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:24.116609 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:24.116934 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:24.116936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:24.116938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:24.117132 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:24.117224 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.117881 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.118004 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:38:24.119107 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.105254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:49513)
0927 12:38:24.106068 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.106127 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.107149 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.107699 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.107706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.108078 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:24.108785 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.108815 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.109664 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.109669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.111079 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.111091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.111815 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.111821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.112069 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.112773 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:24.112796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:24.114250 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:24.116046 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:24.116053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:24.116073 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:24.116417 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:24.116691 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:24.116694 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:24.116696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:24.116841 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:24.117265 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:24.117277 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:24.117583 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.117767 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.118072 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":121,"threads_started":1}
I20260927 12:38:24.120642 16117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34847
I20260927 12:38:24.120896 16117 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:24.121405 16117 heartbeater.cc:511] Master 127.0.98.62:34847 requested a full tablet report, sending...
I20260927 12:38:24.122597 15791 ts_manager.cc:194] Registered new tserver with Master: 52290e478d3e427d953202f45c6d7840 (127.0.98.2:41001)
I20260927 12:38:24.123250 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:38:24.136376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34847
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:24.250979 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:24.251257 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:24.251322 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:24.255669 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:24.255759 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:24.255842 16122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:38:24.261454 16122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:24.262629 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:24.263633 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:24.273346 16130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:24.273376 16128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:24.273336 16127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:24.273586 16122 server_base.cc:1034] running on GCE node
I20260927 12:38:24.274154 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:24.274837 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:24.276046 16122 hybrid_clock.cc:648] HybridClock initialized: now 1790512704275992 us; error 66 us; skew 500 ppm
Sep 27 12:38:24 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:24.279238 16122 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:38:24.280293 16122 webserver.cc:460] Webserver started at http://127.0.98.3:35327/ using document root <none> and password file <none>
I20260927 12:38:24.280864 16122 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:24.280951 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:24.281172 16122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:38:24.283370 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "fa412e36b4266aaaf145faa10dc7b100"
server_key_iv: "6df88fd9af5573ad4bd5830a619a2ca4"
server_key_version: "encryptionkey@0"
I20260927 12:38:24.284163 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "fa412e36b4266aaaf145faa10dc7b100"
server_key_iv: "6df88fd9af5573ad4bd5830a619a2ca4"
server_key_version: "encryptionkey@0"
I20260927 12:38:24.288405 16122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:38:24.291193 16137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:24.292872 16122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:38:24.293044 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "fa412e36b4266aaaf145faa10dc7b100"
server_key_iv: "6df88fd9af5573ad4bd5830a619a2ca4"
server_key_version: "encryptionkey@0"
I20260927 12:38:24.293175 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:24.311128 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:24.315207 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:24.315407 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:24.316028 16122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:24.317256 16122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:24.317400 16122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:24.317483 16122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:24.317512 16122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:24.328197 16122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38013
I20260927 12:38:24.328253 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38013 every 8 connection(s)
I20260927 12:38:24.329136 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:38:24.333598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16122
I20260927 12:38:24.333729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:38:24.334022 392 external_mini_cluster.cc:1428] Setting key d06b041c9e0c4080db6fd08b27ed9b2a
Sep 27 12:38:24 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:24.344803 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.331420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49423 (local address 127.0.98.62:34847)
0927 12:38:24.331593 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.331597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.333315 (+ 1718us) server_negotiation.cc:407] Connection header received
0927 12:38:24.334518 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.334522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.334628 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.334718 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:24.336341 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.336915 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.338116 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.338332 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.338416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.341208 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:24.341211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:24.341213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:24.341241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:24.342683 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:24.343217 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:24.343220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:24.343222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:24.343300 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:24.343632 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:24.343635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:24.343636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:24.343828 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:24.343915 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.344498 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.344631 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 12:38:24.345662 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.332004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:49423)
0927 12:38:24.333068 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.333128 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.334217 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.334940 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.334949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.335444 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:24.336175 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.336198 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.337032 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.337035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.338003 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.338013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.338455 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.338458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.338670 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.339216 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:24.339233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:24.341009 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:24.342809 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:24.342816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:24.342835 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:24.343110 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:24.343380 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:24.343383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:24.343385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:24.343530 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:24.343927 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:24.343935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:24.344253 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.344427 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.344791 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":271,"threads_started":1}
I20260927 12:38:24.346819 16251 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34847
I20260927 12:38:24.347092 16251 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:24.347720 16251 heartbeater.cc:511] Master 127.0.98.62:34847 requested a full tablet report, sending...
I20260927 12:38:24.348935 15791 ts_manager.cc:194] Registered new tserver with Master: a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013)
I20260927 12:38:24.349550 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49423
I20260927 12:38:24.361393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:38:24.371917 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.363467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.0.98.62:34847)
0927 12:38:24.363585 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.363589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.363782 (+ 193us) server_negotiation.cc:407] Connection header received
0927 12:38:24.363843 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.363845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.363892 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.363970 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:24.364824 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.365366 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.366235 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.366470 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.366586 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.368001 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:24.368005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:24.368007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:24.368041 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:24.369696 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:24.370193 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:24.370197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:24.370199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:24.370316 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:24.370835 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:24.370837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:24.370839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:24.371104 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:24.371230 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.371671 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.371788 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:38:24.375191 15791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:38:24.378592 15791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:38:24.396034 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.389244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38013 (local address 127.0.0.1:37612)
0927 12:38:24.390250 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.390307 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.390440 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.391222 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.391228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.391250 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:24.391396 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.391404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.392809 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.392812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.394223 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.394234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.395407 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.395411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.395538 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.395579 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.395690 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.395829 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":79,"threads_started":1}
I20260927 12:38:24.396051 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.389392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.0.98.3:38013)
0927 12:38:24.390732 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.390744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.390767 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:38:24.390845 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.390850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.391067 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.391216 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:24.391529 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.392667 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.394375 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.395292 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.395390 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.395557 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.395734 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.395837 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":115,"threads_started":1}
I20260927 12:38:24.397243 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.390191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43577 (local address 127.0.0.1:43450)
0927 12:38:24.391467 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.391480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.391567 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.392231 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.392234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.392252 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:24.392335 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.392340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.393674 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.393678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.394905 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.394915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.396790 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.396794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.396902 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.396927 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.396997 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.397082 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"mutex_wait_us":81,"thread_start_us":106,"threads_started":1}
I20260927 12:38:24.397583 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.389557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.2:41001)
0927 12:38:24.389907 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.389920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.390824 (+ 904us) server_negotiation.cc:407] Connection header received
0927 12:38:24.390921 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.390927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.391144 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.391302 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:24.391736 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.393979 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.395293 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.396576 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.396676 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.396814 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.397334 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.397449 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260927 12:38:24.397780 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.390381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.98.1:43577)
0927 12:38:24.391048 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.391054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.391656 (+ 602us) server_negotiation.cc:407] Connection header received
0927 12:38:24.391736 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.391741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.391919 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.392059 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:24.392450 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.393543 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.395036 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.396032 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.396529 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.396701 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.397538 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.397652 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":71,"threads_started":1}
I20260927 12:38:24.397400 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.389392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41001 (local address 127.0.0.1:60590)
0927 12:38:24.390702 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.390714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.390816 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.391485 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.391489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.391508 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:24.391604 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.391610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.394150 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.394153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.395127 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.395138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.396865 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.396869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.396943 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.396960 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.397016 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.397155 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":11,"thread_start_us":90,"threads_started":1}
I20260927 12:38:24.399312 16185 tablet_service.cc:1467] Processing CreateTablet for tablet 27d05653961a43d899895eafd642fc75 (DEFAULT_TABLE table=test-table [id=0548c51d4c5e4ed0a8acaf58d63d709f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:24.400401 16185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d05653961a43d899895eafd642fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:24.400622 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 27d05653961a43d899895eafd642fc75 (DEFAULT_TABLE table=test-table [id=0548c51d4c5e4ed0a8acaf58d63d709f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:24.401633 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d05653961a43d899895eafd642fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:24.407694 16268 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Bootstrap starting.
I20260927 12:38:24.408824 16269 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Bootstrap starting.
I20260927 12:38:24.408713 16051 tablet_service.cc:1467] Processing CreateTablet for tablet 27d05653961a43d899895eafd642fc75 (DEFAULT_TABLE table=test-table [id=0548c51d4c5e4ed0a8acaf58d63d709f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:24.409911 16051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d05653961a43d899895eafd642fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:24.410317 16268 tablet_bootstrap.cc:654] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:24.411185 16269 tablet_bootstrap.cc:654] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:24.411690 16268 log.cc:824] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:24.412015 16269 log.cc:824] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:24.413956 16269 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: No bootstrap required, opened a new log
I20260927 12:38:24.414041 16268 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: No bootstrap required, opened a new log
I20260927 12:38:24.414189 16269 ts_tablet_manager.cc:1397] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:38:24.414239 16268 ts_tablet_manager.cc:1397] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:38:24.417841 16269 raft_consensus.cc:348] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.418187 16269 raft_consensus.cc:374] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:24.418246 16269 raft_consensus.cc:729] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8396af3ee64b93b9230c8be15e5b5c, State: Initialized, Role: FOLLOWER
I20260927 12:38:24.418087 16268 raft_consensus.cc:348] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.418366 16268 raft_consensus.cc:374] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:24.418129 16272 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Bootstrap starting.
I20260927 12:38:24.418431 16268 raft_consensus.cc:729] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f90b5bf39b4d6e804ad7dc51ca5194, State: Initialized, Role: FOLLOWER
I20260927 12:38:24.418773 16269 consensus_queue.cc:260] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.419242 16268 consensus_queue.cc:260] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.419883 16269 ts_tablet_manager.cc:1428] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:38:24.420017 15983 heartbeater.cc:503] Master 127.0.98.62:34847 was elected leader, sending a full tablet report...
I20260927 12:38:24.420248 16251 heartbeater.cc:503] Master 127.0.98.62:34847 was elected leader, sending a full tablet report...
I20260927 12:38:24.420264 16268 ts_tablet_manager.cc:1428] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 12:38:24.422309 16272 tablet_bootstrap.cc:654] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:24.423053 16272 log.cc:824] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:24.425321 16272 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: No bootstrap required, opened a new log
I20260927 12:38:24.425520 16272 ts_tablet_manager.cc:1397] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 12:38:24.428781 16272 raft_consensus.cc:348] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.429037 16272 raft_consensus.cc:374] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:24.429071 16272 raft_consensus.cc:729] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52290e478d3e427d953202f45c6d7840, State: Initialized, Role: FOLLOWER
I20260927 12:38:24.429515 16272 consensus_queue.cc:260] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.430263 16117 heartbeater.cc:503] Master 127.0.98.62:34847 was elected leader, sending a full tablet report...
I20260927 12:38:24.430569 16272 ts_tablet_manager.cc:1428] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:38:24.446900 16276 raft_consensus.cc:482] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:24.447122 16276 raft_consensus.cc:504] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.448422 16276 leader_election.cc:288] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013), ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577)
I20260927 12:38:24.452299 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.448734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43577 (local address 127.0.98.2:44533)
0927 12:38:24.448866 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.448881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.448985 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.449266 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.449269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.449286 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:24.449402 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.449408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.450118 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.450121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.451002 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.451012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.451827 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.451830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.451942 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.451968 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.452045 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.452172 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":4480}
I20260927 12:38:24.452332 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.448826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44533 (local address 127.0.98.1:43577)
0927 12:38:24.448962 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.448967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.448984 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:38:24.449132 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.449134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.449179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.449244 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:24.449507 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.450009 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.451147 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.451711 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.451810 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.451880 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.452117 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.452201 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:38:24.452526 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.448735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38013 (local address 127.0.98.2:52149)
0927 12:38:24.449199 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.449210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:24.449309 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:24.449535 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:24.449537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:24.449552 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:24.449668 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.449674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.450408 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.450411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.451326 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:24.451336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.452129 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.452132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:24.452229 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.452252 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:38:24.452325 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:38:24.452392 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":142,"thread_start_us":141,"threads_started":1}
I20260927 12:38:24.452535 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.448814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52149 (local address 127.0.98.3:38013)
0927 12:38:24.448970 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.448975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.449287 (+ 312us) server_negotiation.cc:407] Connection header received
0927 12:38:24.449406 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.449409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.449452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.449515 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:24.449763 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.450278 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.451437 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.451986 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.452073 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.452159 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.452351 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.452427 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:38:24.453053 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d05653961a43d899895eafd642fc75" candidate_uuid: "52290e478d3e427d953202f45c6d7840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" is_pre_election: true
I20260927 12:38:24.453166 16205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d05653961a43d899895eafd642fc75" candidate_uuid: "52290e478d3e427d953202f45c6d7840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" is_pre_election: true
I20260927 12:38:24.453374 15937 raft_consensus.cc:2454] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52290e478d3e427d953202f45c6d7840 in term 0.
I20260927 12:38:24.453437 16205 raft_consensus.cc:2454] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52290e478d3e427d953202f45c6d7840 in term 0.
I20260927 12:38:24.453896 16006 leader_election.cc:302] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52290e478d3e427d953202f45c6d7840, a6f90b5bf39b4d6e804ad7dc51ca5194; no voters:
I20260927 12:38:24.454173 16276 raft_consensus.cc:2781] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:38:24.454259 16276 raft_consensus.cc:482] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:38:24.454298 16276 raft_consensus.cc:3037] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:24.455327 16276 raft_consensus.cc:504] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.455672 16276 leader_election.cc:288] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [CANDIDATE]: Term 1 election: Requested vote from peers a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013), ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577)
I20260927 12:38:24.456067 16205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d05653961a43d899895eafd642fc75" candidate_uuid: "52290e478d3e427d953202f45c6d7840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
I20260927 12:38:24.456080 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d05653961a43d899895eafd642fc75" candidate_uuid: "52290e478d3e427d953202f45c6d7840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed8396af3ee64b93b9230c8be15e5b5c"
I20260927 12:38:24.456238 16205 raft_consensus.cc:3037] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:24.456246 15937 raft_consensus.cc:3037] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:24.457576 15937 raft_consensus.cc:2454] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52290e478d3e427d953202f45c6d7840 in term 1.
I20260927 12:38:24.457576 16205 raft_consensus.cc:2454] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52290e478d3e427d953202f45c6d7840 in term 1.
I20260927 12:38:24.457932 16006 leader_election.cc:302] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52290e478d3e427d953202f45c6d7840, a6f90b5bf39b4d6e804ad7dc51ca5194, ed8396af3ee64b93b9230c8be15e5b5c; no voters:
I20260927 12:38:24.458125 16276 raft_consensus.cc:2781] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:24.458370 16276 raft_consensus.cc:686] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 LEADER]: Becoming Leader. State: Replica: 52290e478d3e427d953202f45c6d7840, State: Running, Role: LEADER
I20260927 12:38:24.458770 16276 consensus_queue.cc:237] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:24.462213 15791 catalog_manager.cc:5495] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52290e478d3e427d953202f45c6d7840 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "52290e478d3e427d953202f45c6d7840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } health_report { overall_health: UNKNOWN } } }
I20260927 12:38:24.476830 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.473184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.98.2:41001)
0927 12:38:24.473313 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:24.473318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:24.473331 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:38:24.473407 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:24.473410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:24.473464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:24.473573 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:38:24.473833 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.474340 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.475261 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:24.475487 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:24.475585 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:24.475875 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:38:24.476348 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:38:24.476457 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:24.476517 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 12:38:24.476600 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:38:24.483184 15937 raft_consensus.cc:1261] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Refusing update from remote peer 52290e478d3e427d953202f45c6d7840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:24.483184 16205 raft_consensus.cc:1261] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Refusing update from remote peer 52290e478d3e427d953202f45c6d7840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:24.484215 16278 consensus_queue.cc:1035] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:24.484416 16276 consensus_queue.cc:1035] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:24.494853 16283 mvcc.cc:205] Tried to move back new op lower bound from 7333940037555572736 to 7333940037465661440. Current Snapshot: MvccSnapshot[applied={T|T < 7333940037555572736}]
I20260927 12:38:24.504063 15792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:24.504352 15792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:24.507073 15792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Sending DeleteTablet for 3 replicas of tablet 27d05653961a43d899895eafd642fc75
I20260927 12:38:24.508080 16185 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d05653961a43d899895eafd642fc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37612
I20260927 12:38:24.508239 16051 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d05653961a43d899895eafd642fc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60590
I20260927 12:38:24.508354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15854
I20260927 12:38:24.510165 16292 tablet_replica.cc:330] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: stopping tablet replica
I20260927 12:38:24.510587 16292 raft_consensus.cc:2229] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:24.510985 16292 raft_consensus.cc:2258] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:24.511441 16291 tablet_replica.cc:330] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: stopping tablet replica
I20260927 12:38:24.511790 16291 raft_consensus.cc:2229] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:38:24.512320 16291 raft_consensus.cc:2258] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:24.513222 16292 ts_tablet_manager.cc:1905] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:24.513826 16291 ts_tablet_manager.cc:1905] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:24.514392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15988
W20260927 12:38:24.514422 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43577 (error 108)
I20260927 12:38:24.516011 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.515694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43577 (local address 127.0.0.1:43454)
0927 12:38:24.515846 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.515920 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 12:38:24.516184 15776 catalog_manager.cc:4584] TS ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27d05653961a43d899895eafd642fc75: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43577: connect: Connection refused (error 111)
I20260927 12:38:24.518049 16292 ts_tablet_manager.cc:1918] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:24.518162 16292 log.cc:1196] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/27d05653961a43d899895eafd642fc75
I20260927 12:38:24.518460 16292 ts_tablet_manager.cc:1939] T 27d05653961a43d899895eafd642fc75 P a6f90b5bf39b4d6e804ad7dc51ca5194: Deleting consensus metadata
I20260927 12:38:24.519912 15778 catalog_manager.cc:4841] TS a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013): tablet 27d05653961a43d899895eafd642fc75 (table test-table [id=0548c51d4c5e4ed0a8acaf58d63d709f]) successfully deleted
I20260927 12:38:24.520648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 16122
I20260927 12:38:24.521641 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:24.520697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41001 (local address 127.0.0.1:60620)
0927 12:38:24.521481 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:24.521549 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":699}
I20260927 12:38:24.525846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 15760
I20260927 12:38:44.532308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--webserver_interface=127.0.98.62
--webserver_port=37895
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:38:44.669611 16294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:44.669965 16294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:44.670014 16294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:44.674502 16294 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:38:44.674587 16294 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:38:44.674602 16294 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:38:44.674615 16294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:38:44.674635 16294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:44.674654 16294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:44.674669 16294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:38:44.674685 16294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:38:44.680800 16294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:44.682243 16294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:44.683559 16294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:44.700026 16300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:44.700184 16301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:44.700165 16294 server_base.cc:1034] running on GCE node
W20260927 12:38:44.700048 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:44.701251 16294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:44.702749 16294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:44.703986 16294 hybrid_clock.cc:648] HybridClock initialized: now 1790512724703945 us; error 55 us; skew 500 ppm
Sep 27 12:38:44 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:44.714419 16294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:38:44.715690 16294 webserver.cc:460] Webserver started at http://127.0.98.62:37895/ using document root <none> and password file <none>
I20260927 12:38:44.716459 16294 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:44.716559 16294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:44.721601 16294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:38:44.724658 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:44.726455 16294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 12:38:44.726641 16294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "7b1fa084526d4d4585c562f7b5c938c6"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "1aab24d57442702218edeb97dcb4c1d4"
server_key_iv: "b3aaa840a0fbc9abdd3d0f8ce1c455dd"
server_key_version: "encryptionkey@0"
I20260927 12:38:44.727358 16294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:44.743497 16294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:44.747035 16294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:44.747238 16294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:44.755833 16294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34847
I20260927 12:38:44.756215 16362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34847 every 8 connection(s)
I20260927 12:38:44.757239 16294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:38:44.761842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16294
I20260927 12:38:44.762274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43577
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34847
--webserver_port=45309
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:44.763185 16363 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:38:44.765516 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap starting.
I20260927 12:38:44.786062 16363 log.cc:824] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:44.796793 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:44.797247 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap complete.
I20260927 12:38:44.802062 16363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:44.802608 16363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1fa084526d4d4585c562f7b5c938c6, State: Initialized, Role: FOLLOWER
I20260927 12:38:44.803382 16363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:44.803527 16363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:38:44.803604 16363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:38:44.803716 16363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:38:44.806260 16363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:44.806685 16363 leader_election.cc:302] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b1fa084526d4d4585c562f7b5c938c6; no voters:
I20260927 12:38:44.807065 16363 leader_election.cc:288] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:38:44.807209 16367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:38:44.807451 16367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 LEADER]: Becoming Leader. State: Replica: 7b1fa084526d4d4585c562f7b5c938c6, State: Running, Role: LEADER
I20260927 12:38:44.807899 16367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:38:44.808717 16363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:38:44.809840 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7b1fa084526d4d4585c562f7b5c938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } }
I20260927 12:38:44.810112 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:44.810766 16376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:38:44.810847 16369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b1fa084526d4d4585c562f7b5c938c6. Latest consensus state: current_term: 2 leader_uuid: "7b1fa084526d4d4585c562f7b5c938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } }
I20260927 12:38:44.811028 16369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:38:44.817444 16376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:38:44.818332 16376 catalog_manager.cc:1259] Loaded cluster ID: 17d5a043628c444595efa6ed071c30a4
I20260927 12:38:44.818400 16376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:38:44.820775 16376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:38:44.821504 16376 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Loaded TSK: 0
I20260927 12:38:44.833096 16376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Generated new TSK 1
I20260927 12:38:44.833449 16376 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:38:44.908321 16365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:44.908710 16365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:44.908778 16365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:44.913362 16365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:44.913465 16365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:44.913534 16365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:38:44.919714 16365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45309
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:44.921229 16365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:44.922487 16365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:44.934540 16391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:44.934613 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:44.934979 16365 server_base.cc:1034] running on GCE node
W20260927 12:38:44.934540 16393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:44.935814 16365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:44.936908 16365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:44.938159 16365 hybrid_clock.cc:648] HybridClock initialized: now 1790512724938119 us; error 65 us; skew 500 ppm
Sep 27 12:38:44 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:44.942317 16365 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:38:44.943672 16365 webserver.cc:460] Webserver started at http://127.0.98.1:45309/ using document root <none> and password file <none>
I20260927 12:38:44.944419 16365 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:44.944511 16365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:44.949731 16365 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:38:44.952982 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:44.954828 16365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:38:44.954984 16365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "ed8396af3ee64b93b9230c8be15e5b5c"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "066e0cbbbdbc9cc917a86f2dc85a5dfa"
server_key_iv: "ac017d09319f61453c6219256cacc165"
server_key_version: "encryptionkey@0"
I20260927 12:38:44.955736 16365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:44.973814 16365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:44.978122 16365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:44.978353 16365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:44.979084 16365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:44.980608 16407 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:38:44.985793 16365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:38:44.985924 16365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 12:38:44.986016 16365 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:38:44.988690 16365 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:38:44.988761 16365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:44.989296 16407 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Bootstrap starting.
I20260927 12:38:45.002225 16365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43577
I20260927 12:38:45.002705 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43577 every 8 connection(s)
I20260927 12:38:45.003969 16365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:38:45.009765 16407 log.cc:824] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:45.010834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16365
I20260927 12:38:45.011394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41001
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34847
--webserver_port=41603
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:45.018915 16407 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:45.019375 16407 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Bootstrap complete.
Sep 27 12:38:45 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:45.020545 16407 ts_tablet_manager.cc:1397] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20260927 12:38:45.023895 16407 raft_consensus.cc:348] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:45.024536 16407 raft_consensus.cc:729] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8396af3ee64b93b9230c8be15e5b5c, State: Initialized, Role: FOLLOWER
I20260927 12:38:45.025125 16407 consensus_queue.cc:260] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:45.026243 16407 ts_tablet_manager.cc:1428] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:38:45.028635 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.007001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:49639)
0927 12:38:45.008396 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.008467 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.009576 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.011298 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.011310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.011846 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:45.012656 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.012682 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.014307 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.014311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.015759 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.015774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.016871 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.016877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.017124 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.017884 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:45.017908 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:45.020118 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:45.024406 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.024417 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.024439 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.024890 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.025446 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.025450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.025452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.025658 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.026711 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:45.026719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:45.027077 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.027307 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.027657 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":173,"threads_started":1}
I20260927 12:38:45.029213 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.007090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49639 (local address 127.0.98.62:34847)
0927 12:38:45.008400 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.008424 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.008621 (+ 197us) server_negotiation.cc:407] Connection header received
0927 12:38:45.010105 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.010118 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.010562 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.011023 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:45.012936 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.014116 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.015949 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.016632 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.016868 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.020487 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:45.020497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:45.020520 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:45.020576 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:45.024125 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.025084 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.025092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.025100 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.025317 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.025786 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.025790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.025792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.026448 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:45.026604 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.027562 (+ 958us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.028027 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":175,"threads_started":1}
I20260927 12:38:45.032220 16515 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34847
I20260927 12:38:45.032544 16515 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:45.033198 16515 heartbeater.cc:511] Master 127.0.98.62:34847 requested a full tablet report, sending...
I20260927 12:38:45.036347 16321 ts_manager.cc:194] Registered new tserver with Master: ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577)
I20260927 12:38:45.040252 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49639
I20260927 12:38:45.043519 16515 heartbeater.cc:503] Master 127.0.98.62:34847 was elected leader, sending a full tablet report...
Sep 27 12:38:45 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:38:45.055321 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.039969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43577 (local address 127.0.0.1:34118)
0927 12:38:45.041043 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.041097 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.041293 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.041955 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.041963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.042496 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:45.042700 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.042710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.044061 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.044066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.044849 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.044860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.045471 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.045493 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.045612 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.046590 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:45.046606 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:45.049320 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:45.052066 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.052082 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.052208 (+ 126us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.052710 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.053259 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.053264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.053266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.053501 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.054464 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:45.054473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:45.054663 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.054805 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.055098 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":200,"threads_started":1}
I20260927 12:38:45.055344 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.040157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.98.1:43577)
0927 12:38:45.041125 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.041138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.041246 (+ 108us) server_negotiation.cc:407] Connection header received
0927 12:38:45.041501 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.041507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.041749 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.041901 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:45.042903 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.043884 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.044999 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.045357 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.045456 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.049592 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:45.049606 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:45.049621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:45.049653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:45.051826 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.052932 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.052939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.052940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.053058 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.053808 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.053813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.053815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.054191 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:45.054480 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.054950 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.055194 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":156,"threads_started":1}
I20260927 12:38:45.056862 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d05653961a43d899895eafd642fc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34118
I20260927 12:38:45.058223 16524 tablet_replica.cc:330] stopping tablet replica
I20260927 12:38:45.058590 16524 raft_consensus.cc:2229] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:45.058862 16524 raft_consensus.cc:2258] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:45.060078 16524 ts_tablet_manager.cc:1905] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:45.065536 16524 ts_tablet_manager.cc:1918] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:45.065661 16524 log.cc:1196] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/27d05653961a43d899895eafd642fc75
I20260927 12:38:45.065909 16524 ts_tablet_manager.cc:1939] T 27d05653961a43d899895eafd642fc75 P ed8396af3ee64b93b9230c8be15e5b5c: Deleting consensus metadata
I20260927 12:38:45.068543 16311 catalog_manager.cc:4841] TS ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577): tablet 27d05653961a43d899895eafd642fc75 (table test-table [id=0548c51d4c5e4ed0a8acaf58d63d709f]) successfully deleted
W20260927 12:38:45.157104 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:45.157481 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:45.157532 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:45.162145 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:45.162259 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:45.162331 16520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:38:45.169442 16520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41603
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:45.171137 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:45.172508 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:45.185045 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:45.185081 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:45.185045 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:45.185531 16520 server_base.cc:1034] running on GCE node
I20260927 12:38:45.186199 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:45.187125 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:45.188359 16520 hybrid_clock.cc:648] HybridClock initialized: now 1790512725188313 us; error 63 us; skew 500 ppm
Sep 27 12:38:45 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:45.191917 16520 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:38:45.193182 16520 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260927 12:38:45.193921 16520 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:45.194013 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:45.199230 16520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:38:45.202425 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:45.204744 16520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 12:38:45.204926 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "52290e478d3e427d953202f45c6d7840"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "de4f5f8410329b8ec53af0bcf93a7e29"
server_key_iv: "e1b762e24544c276d665a6fe3b81a699"
server_key_version: "encryptionkey@0"
I20260927 12:38:45.205626 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:45.219712 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:45.223255 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:45.223443 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:45.224184 16520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:45.225512 16546 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:38:45.231132 16520 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:38:45.231251 16520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 12:38:45.231333 16520 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:38:45.234217 16520 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:38:45.234300 16520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260927 12:38:45.234876 16546 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Bootstrap starting.
I20260927 12:38:45.243772 16546 log.cc:824] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:45.246284 16520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41001
I20260927 12:38:45.246351 16654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41001 every 8 connection(s)
I20260927 12:38:45.247571 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:38:45.255091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16520
I20260927 12:38:45.255237 16546 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:38:45.255676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38013
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34847
--webserver_port=35327
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:38:45.255744 16546 tablet_bootstrap.cc:492] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Bootstrap complete.
I20260927 12:38:45.256469 16546 ts_tablet_manager.cc:1397] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20260927 12:38:45.260660 16546 raft_consensus.cc:348] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:45.261329 16546 raft_consensus.cc:729] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52290e478d3e427d953202f45c6d7840, State: Initialized, Role: FOLLOWER
Sep 27 12:38:45 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:45.262754 16546 consensus_queue.cc:260] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } }
I20260927 12:38:45.263988 16546 ts_tablet_manager.cc:1428] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:38:45.268039 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.250146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44765 (local address 127.0.98.62:34847)
0927 12:38:45.250316 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.250321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.251223 (+ 902us) server_negotiation.cc:407] Connection header received
0927 12:38:45.253239 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.253244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.253322 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.253398 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:45.255259 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.256072 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.258105 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.258616 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.258730 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.263278 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:45.263284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:45.263287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:45.263344 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:45.265453 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.266112 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.266116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.266118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.266235 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.266672 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.266675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.266677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.266949 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:45.267080 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.267693 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.267853 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:38:45.269106 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.250348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:44765)
0927 12:38:45.250969 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.251032 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.252916 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.253640 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.253650 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.254074 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:45.254951 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.254980 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.256324 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.256329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.257872 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.257888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.259027 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.259031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.259277 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.260804 (+ 1527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:45.260828 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:45.262898 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:45.265640 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.265646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.265660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.265974 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.266381 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.266384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.266385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.266548 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.267097 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:45.267104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:45.267418 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.267613 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.267991 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":128,"threads_started":1}
I20260927 12:38:45.270743 16655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34847
I20260927 12:38:45.271138 16655 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:45.271838 16655 heartbeater.cc:511] Master 127.0.98.62:34847 requested a full tablet report, sending...
I20260927 12:38:45.274264 16321 ts_manager.cc:194] Registered new tserver with Master: 52290e478d3e427d953202f45c6d7840 (127.0.98.2:41001)
I20260927 12:38:45.275626 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44765
I20260927 12:38:45.278060 16655 heartbeater.cc:503] Master 127.0.98.62:34847 was elected leader, sending a full tablet report...
I20260927 12:38:45.282380 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.275319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41001 (local address 127.0.0.1:39310)
0927 12:38:45.275533 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.275548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.275631 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.277399 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.277402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.277433 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:45.277566 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.277573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.279731 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.279734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.280773 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.280784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.281950 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.281953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.282043 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.282082 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.282137 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.282261 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260927 12:38:45.282464 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.275844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.98.2:41001)
0927 12:38:45.276401 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.276414 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.276445 (+ 31us) server_negotiation.cc:407] Connection header received
0927 12:38:45.276545 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.276552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.277169 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.277735 (+ 566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:45.277881 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.279378 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.280925 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.281796 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.281932 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.282112 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.282172 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.282273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":159,"threads_started":1}
I20260927 12:38:45.283710 16588 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d05653961a43d899895eafd642fc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39310
I20260927 12:38:45.285068 16661 tablet_replica.cc:330] stopping tablet replica
I20260927 12:38:45.285405 16661 raft_consensus.cc:2229] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:45.285688 16661 raft_consensus.cc:2258] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:45.286882 16661 ts_tablet_manager.cc:1905] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:45.291374 16661 ts_tablet_manager.cc:1918] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:45.291501 16661 log.cc:1196] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/27d05653961a43d899895eafd642fc75
I20260927 12:38:45.291805 16661 ts_tablet_manager.cc:1939] T 27d05653961a43d899895eafd642fc75 P 52290e478d3e427d953202f45c6d7840: Deleting consensus metadata
I20260927 12:38:45.294021 16311 catalog_manager.cc:4841] TS 52290e478d3e427d953202f45c6d7840 (127.0.98.2:41001): tablet 27d05653961a43d899895eafd642fc75 (table test-table [id=0548c51d4c5e4ed0a8acaf58d63d709f]) successfully deleted
W20260927 12:38:45.385809 16658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:38:45.386118 16658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:38:45.386183 16658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:38:45.390821 16658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:38:45.390928 16658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:38:45.391018 16658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:38:45.397477 16658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35327
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:38:45.398716 16658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:38:45.399796 16658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:38:45.410472 16666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:45.410403 16669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:38:45.410398 16667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:38:45.410980 16658 server_base.cc:1034] running on GCE node
I20260927 12:38:45.411481 16658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:38:45.412268 16658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:38:45.413504 16658 hybrid_clock.cc:648] HybridClock initialized: now 1790512725413463 us; error 67 us; skew 500 ppm
Sep 27 12:38:45 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:38:45.416823 16658 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:38:45.417912 16658 webserver.cc:460] Webserver started at http://127.0.98.3:35327/ using document root <none> and password file <none>
I20260927 12:38:45.418519 16658 fs_manager.cc:359] Metadata directory not provided
I20260927 12:38:45.418620 16658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:38:45.423102 16658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 12:38:45.425583 16676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:45.427062 16658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:38:45.427210 16658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "fa412e36b4266aaaf145faa10dc7b100"
server_key_iv: "6df88fd9af5573ad4bd5830a619a2ca4"
server_key_version: "encryptionkey@0"
I20260927 12:38:45.427827 16658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:38:45.471138 16658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:38:45.474428 16658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:38:45.474617 16658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:38:45.475227 16658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:38:45.476059 16658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:38:45.476135 16658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:45.476191 16658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:38:45.476220 16658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:38:45.484921 16658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38013
I20260927 12:38:45.484978 16789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38013 every 8 connection(s)
I20260927 12:38:45.485948 16658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:38:45.494155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16658
Sep 27 12:38:45 dist-test-slave-tzgx krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:38:45.500679 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.487870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43303 (local address 127.0.98.62:34847)
0927 12:38:45.488004 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.488007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.489138 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 12:38:45.490014 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.490017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.490068 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.490149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:45.491352 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.491949 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.493257 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.493529 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.493627 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.496528 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:45.496531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:45.496533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:45.496556 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:45.498300 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.498957 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.498960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.498961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.499048 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.499439 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.499442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.499443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.499634 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:45.499723 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.500366 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.500504 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:38:45.501633 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.488270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:43303)
0927 12:38:45.488975 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.489021 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.489817 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.490294 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.490300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.490671 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:38:45.491186 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.491201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.492128 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.492133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.493118 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.493129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.493682 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.493685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.493870 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.494572 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:38:45.494592 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:38:45.496363 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:38:45.498414 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.498421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.498440 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.498808 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.499145 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:38:45.499148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:38:45.499150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:38:45.499317 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:38:45.499766 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:38:45.499772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:38:45.500040 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.500278 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.500632 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":118,"threads_started":1}
I20260927 12:38:45.503034 16790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34847
I20260927 12:38:45.503302 16790 heartbeater.cc:464] Registering TS with master...
I20260927 12:38:45.503775 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.494895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.0.98.62:34847)
0927 12:38:45.495179 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.495183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.495198 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:38:45.495341 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.495345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.495386 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.495462 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:45.496354 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.497021 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.498277 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.498512 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.498636 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.500200 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:45.500203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:45.500205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:45.500225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:45.501912 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.502507 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.502510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.502512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.502606 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.502903 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.502907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.502909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.503140 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:45.503336 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.503492 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.503619 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260927 12:38:45.503876 16790 heartbeater.cc:511] Master 127.0.98.62:34847 requested a full tablet report, sending...
I20260927 12:38:45.505683 16321 ts_manager.cc:194] Registered new tserver with Master: a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013)
I20260927 12:38:45.506378 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43303
I20260927 12:38:45.518006 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:38:45.522099 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.518693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.0.98.62:34847)
0927 12:38:45.518827 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.518831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.518996 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:38:45.519154 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.519156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.519262 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.519351 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:38:45.519553 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.520076 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.520939 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.521184 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.521261 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.521597 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:38:45.521745 (+ 148us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 12:38:45.521965 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50972: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 12:38:45.522169 16794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50972: token expired
W20260927 12:38:45.522639 16255 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34847)
I20260927 12:38:45.529914 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.523005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.0.98.62:34847)
0927 12:38:45.523123 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.523127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.523138 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:38:45.523228 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.523230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.523267 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.523328 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:38:45.523921 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.524520 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.525315 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.525543 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.525628 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.526930 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:38:45.526934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:38:45.526936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:38:45.526958 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:38:45.528318 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.528748 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.528750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.528752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.528825 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:38:45.529116 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:38:45.529119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:38:45.529120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:38:45.529299 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:38:45.529396 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.529662 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.529762 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:38:45.532465 16321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:38:45.535992 16321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:38:45.543965 16449 tablet_service.cc:1467] Processing CreateTablet for tablet f3885a1f1a354a8089663d4039e7793e (DEFAULT_TABLE table=test-table [id=e8cb750677984f3897a3aa5ac967b8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:45.543965 16588 tablet_service.cc:1467] Processing CreateTablet for tablet f3885a1f1a354a8089663d4039e7793e (DEFAULT_TABLE table=test-table [id=e8cb750677984f3897a3aa5ac967b8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:45.544626 16588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3885a1f1a354a8089663d4039e7793e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:45.544626 16449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3885a1f1a354a8089663d4039e7793e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:45.547454 16546 tablet_bootstrap.cc:492] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Bootstrap starting.
I20260927 12:38:45.548230 16800 tablet_bootstrap.cc:492] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Bootstrap starting.
I20260927 12:38:45.548530 16546 tablet_bootstrap.cc:654] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:45.548638 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.543208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38013 (local address 127.0.0.1:53034)
0927 12:38:45.543328 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.543345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.543420 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.544479 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.544482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.544501 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:45.544618 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.544624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.545849 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.545853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.546938 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.546949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.548219 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.548223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.548322 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.548350 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.548403 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.548492 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:38:45.549321 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.543589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.0.98.3:38013)
0927 12:38:45.543941 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.543954 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.543975 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:38:45.544041 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.544045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.544308 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.544513 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:45.544714 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.545750 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.547083 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.548048 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.548822 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.549003 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.549061 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.549173 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":148,"threads_started":1}
I20260927 12:38:45.549732 16546 tablet_bootstrap.cc:492] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: No bootstrap required, opened a new log
I20260927 12:38:45.549817 16546 ts_tablet_manager.cc:1397] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:38:45.549957 16800 tablet_bootstrap.cc:654] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:45.550561 16546 raft_consensus.cc:348] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.550690 16546 raft_consensus.cc:374] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:45.550727 16546 raft_consensus.cc:729] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52290e478d3e427d953202f45c6d7840, State: Initialized, Role: FOLLOWER
I20260927 12:38:45.550855 16546 consensus_queue.cc:260] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.551105 16546 ts_tablet_manager.cc:1428] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:38:45.551186 16800 tablet_bootstrap.cc:492] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: No bootstrap required, opened a new log
I20260927 12:38:45.551280 16800 ts_tablet_manager.cc:1397] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:38:45.551640 16800 raft_consensus.cc:348] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.551736 16800 raft_consensus.cc:374] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:45.551772 16800 raft_consensus.cc:729] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8396af3ee64b93b9230c8be15e5b5c, State: Initialized, Role: FOLLOWER
I20260927 12:38:45.551896 16800 consensus_queue.cc:260] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.552673 16800 ts_tablet_manager.cc:1428] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:38:45.553536 16724 tablet_service.cc:1467] Processing CreateTablet for tablet f3885a1f1a354a8089663d4039e7793e (DEFAULT_TABLE table=test-table [id=e8cb750677984f3897a3aa5ac967b8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:38:45.554757 16724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3885a1f1a354a8089663d4039e7793e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:38:45.561866 16804 tablet_bootstrap.cc:492] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Bootstrap starting.
I20260927 12:38:45.564015 16804 tablet_bootstrap.cc:654] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Neither blocks nor log segments found. Creating new log.
I20260927 12:38:45.565114 16804 log.cc:824] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Log is configured to *not* fsync() on all Append() calls
I20260927 12:38:45.567211 16804 tablet_bootstrap.cc:492] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: No bootstrap required, opened a new log
I20260927 12:38:45.567421 16804 ts_tablet_manager.cc:1397] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:38:45.570420 16804 raft_consensus.cc:348] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.570667 16804 raft_consensus.cc:374] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:38:45.570704 16804 raft_consensus.cc:729] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f90b5bf39b4d6e804ad7dc51ca5194, State: Initialized, Role: FOLLOWER
I20260927 12:38:45.571103 16804 consensus_queue.cc:260] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.571736 16790 heartbeater.cc:503] Master 127.0.98.62:34847 was elected leader, sending a full tablet report...
I20260927 12:38:45.571982 16804 ts_tablet_manager.cc:1428] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 12:38:45.687958 16803 raft_consensus.cc:482] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:38:45.688223 16803 raft_consensus.cc:504] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.689307 16803 leader_election.cc:288] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013), 52290e478d3e427d953202f45c6d7840 (127.0.98.2:41001)
I20260927 12:38:45.693041 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.689673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35793 (local address 127.0.98.2:41001)
0927 12:38:45.689798 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.689802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.689991 (+ 189us) server_negotiation.cc:407] Connection header received
0927 12:38:45.690156 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.690158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.690200 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.690260 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:45.690512 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.691003 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.692027 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.692482 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.692578 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.692642 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.692838 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.692919 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:38:45.693042 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.689709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58535 (local address 127.0.98.3:38013)
0927 12:38:45.689874 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.689879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.690100 (+ 221us) server_negotiation.cc:407] Connection header received
0927 12:38:45.690234 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.690238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.690281 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.690342 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:38:45.690598 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.691100 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.692068 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.692504 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.692606 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.692665 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.692835 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.692911 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:38:45.693073 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.689569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41001 (local address 127.0.98.1:35793)
0927 12:38:45.689895 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.689908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.690029 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.690289 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.690291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.690307 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:45.690410 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.690416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.691108 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.691111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.691899 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.691907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.692607 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.692610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.692694 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.692739 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.692804 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.692918 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260927 12:38:45.693079 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.689587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38013 (local address 127.0.98.1:58535)
0927 12:38:45.690007 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.690018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:38:45.690131 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:38:45.690371 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:38:45.690374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:38:45.690389 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:38:45.690487 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.690492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.691212 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.691215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.691943 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:38:45.691951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.692625 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.692628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:38:45.692726 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.692744 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:38:45.692808 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:38:45.692922 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":24,"thread_start_us":64,"threads_started":1}
I20260927 12:38:45.693684 16608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3885a1f1a354a8089663d4039e7793e" candidate_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52290e478d3e427d953202f45c6d7840" is_pre_election: true
I20260927 12:38:45.693931 16744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3885a1f1a354a8089663d4039e7793e" candidate_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" is_pre_election: true
I20260927 12:38:45.693986 16608 raft_consensus.cc:2454] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed8396af3ee64b93b9230c8be15e5b5c in term 0.
I20260927 12:38:45.694199 16744 raft_consensus.cc:2454] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed8396af3ee64b93b9230c8be15e5b5c in term 0.
I20260927 12:38:45.694398 16401 leader_election.cc:302] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52290e478d3e427d953202f45c6d7840, ed8396af3ee64b93b9230c8be15e5b5c; no voters:
I20260927 12:38:45.694694 16803 raft_consensus.cc:2781] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:38:45.694772 16803 raft_consensus.cc:482] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:38:45.694802 16803 raft_consensus.cc:3037] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:45.696055 16803 raft_consensus.cc:504] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.696471 16803 leader_election.cc:288] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [CANDIDATE]: Term 1 election: Requested vote from peers a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013), 52290e478d3e427d953202f45c6d7840 (127.0.98.2:41001)
I20260927 12:38:45.696812 16744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3885a1f1a354a8089663d4039e7793e" candidate_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
I20260927 12:38:45.696909 16608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3885a1f1a354a8089663d4039e7793e" candidate_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52290e478d3e427d953202f45c6d7840"
I20260927 12:38:45.696936 16744 raft_consensus.cc:3037] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:45.697053 16608 raft_consensus.cc:3037] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:38:45.698028 16744 raft_consensus.cc:2454] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed8396af3ee64b93b9230c8be15e5b5c in term 1.
I20260927 12:38:45.698120 16608 raft_consensus.cc:2454] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed8396af3ee64b93b9230c8be15e5b5c in term 1.
I20260927 12:38:45.698351 16403 leader_election.cc:302] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6f90b5bf39b4d6e804ad7dc51ca5194, ed8396af3ee64b93b9230c8be15e5b5c; no voters:
I20260927 12:38:45.698544 16803 raft_consensus.cc:2781] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:38:45.698728 16803 raft_consensus.cc:686] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 1 LEADER]: Becoming Leader. State: Replica: ed8396af3ee64b93b9230c8be15e5b5c, State: Running, Role: LEADER
I20260927 12:38:45.698990 16803 consensus_queue.cc:237] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } }
I20260927 12:38:45.702289 16321 catalog_manager.cc:5495] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed8396af3ee64b93b9230c8be15e5b5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 } health_report { overall_health: UNKNOWN } } }
W20260927 12:38:45.736938 16791 tablet.cc:2369] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:38:45.742815 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.738447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.0.98.1:43577)
0927 12:38:45.738772 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 12:38:45.738779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:38:45.738793 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:38:45.738869 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:38:45.738872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:38:45.738926 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:38:45.739031 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:38:45.739350 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.739944 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.740871 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:38:45.741580 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:38:45.741673 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:38:45.741955 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:38:45.742409 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:38:45.742531 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 12:38:45.742620 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 12:38:45.742676 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":148,"threads_started":1}
I20260927 12:38:45.747248 16608 raft_consensus.cc:1261] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Refusing update from remote peer ed8396af3ee64b93b9230c8be15e5b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:45.747252 16744 raft_consensus.cc:1261] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Refusing update from remote peer ed8396af3ee64b93b9230c8be15e5b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:38:45.748075 16803 consensus_queue.cc:1035] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "52290e478d3e427d953202f45c6d7840" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:38:45.748324 16809 consensus_queue.cc:1035] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:38:45.748975 16656 tablet.cc:2369] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:38:45.755427 16813 mvcc.cc:205] Tried to move back new op lower bound from 7333940124653043712 to 7333940124465557504. Current Snapshot: MvccSnapshot[applied={T|T < 7333940124653043712}]
I20260927 12:38:45.765508 16325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:45.765703 16325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:38:45.770612 16325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Sending DeleteTablet for 3 replicas of tablet f3885a1f1a354a8089663d4039e7793e
I20260927 12:38:45.771646 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet f3885a1f1a354a8089663d4039e7793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34118
I20260927 12:38:45.771988 16822 tablet_replica.cc:330] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: stopping tablet replica
I20260927 12:38:45.772147 16588 tablet_service.cc:1514] Processing DeleteTablet for tablet f3885a1f1a354a8089663d4039e7793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39310
I20260927 12:38:45.772311 16661 tablet_replica.cc:330] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: stopping tablet replica
I20260927 12:38:45.772365 16822 raft_consensus.cc:2229] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:38:45.772414 16661 raft_consensus.cc:2229] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:45.772531 16661 raft_consensus.cc:2258] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:45.772677 16822 raft_consensus.cc:2258] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:45.773365 16822 ts_tablet_manager.cc:1905] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:45.773471 16724 tablet_service.cc:1514] Processing DeleteTablet for tablet f3885a1f1a354a8089663d4039e7793e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:38:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53034
I20260927 12:38:45.774153 16823 tablet_replica.cc:330] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: stopping tablet replica
I20260927 12:38:45.774386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 16365
I20260927 12:38:45.775103 16823 raft_consensus.cc:2229] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:38:45.775492 16823 raft_consensus.cc:2258] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:38:45.777746 16823 ts_tablet_manager.cc:1905] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:45.778581 16661 ts_tablet_manager.cc:1905] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:38:45.780279 16823 ts_tablet_manager.cc:1918] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:45.780372 16823 log.cc:1196] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/f3885a1f1a354a8089663d4039e7793e
I20260927 12:38:45.780678 16823 ts_tablet_manager.cc:1939] T f3885a1f1a354a8089663d4039e7793e P a6f90b5bf39b4d6e804ad7dc51ca5194: Deleting consensus metadata
I20260927 12:38:45.780964 16661 ts_tablet_manager.cc:1918] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:38:45.781036 16661 log.cc:1196] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/f3885a1f1a354a8089663d4039e7793e
I20260927 12:38:45.781250 16661 ts_tablet_manager.cc:1939] T f3885a1f1a354a8089663d4039e7793e P 52290e478d3e427d953202f45c6d7840: Deleting consensus metadata
W20260927 12:38:45.781551 16542 connection.cc:535] server connection from 127.0.98.1:35793 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 12:38:45.781903 16311 connection.cc:535] server connection from 127.0.98.1:49639 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 12:38:45.782166 16311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43577 (error 108)
I20260927 12:38:45.782249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 16520
I20260927 12:38:45.782625 16311 catalog_manager.cc:4841] TS 52290e478d3e427d953202f45c6d7840 (127.0.98.2:41001): tablet f3885a1f1a354a8089663d4039e7793e (table test-table [id=e8cb750677984f3897a3aa5ac967b8b4]) successfully deleted
I20260927 12:38:45.783010 16313 catalog_manager.cc:4841] TS a6f90b5bf39b4d6e804ad7dc51ca5194 (127.0.98.3:38013): tablet f3885a1f1a354a8089663d4039e7793e (table test-table [id=e8cb750677984f3897a3aa5ac967b8b4]) successfully deleted
I20260927 12:38:45.784976 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:38:45.784668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43577 (local address 127.0.0.1:34138)
0927 12:38:45.784829 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 12:38:45.784900 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 12:38:45.785143 16311 catalog_manager.cc:4584] TS ed8396af3ee64b93b9230c8be15e5b5c (127.0.98.1:43577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3885a1f1a354a8089663d4039e7793e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43577: connect: Connection refused (error 111)
I20260927 12:38:45.788071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 16658
I20260927 12:38:45.795766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 16294
I20260927 12:39:05.802366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--webserver_interface=127.0.98.62
--webserver_port=37895
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:05.913493 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:05.913815 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:05.913856 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:05.918678 16825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:39:05.918762 16825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:39:05.918782 16825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:39:05.918802 16825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:05.918829 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:05.918862 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:05.918893 16825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:05.918922 16825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:05.927383 16825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:05.929708 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:05.931118 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:05.941553 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:05.941696 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:05.941593 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:05.941586 16825 server_base.cc:1034] running on GCE node
I20260927 12:39:05.942493 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:05.943675 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:05.944862 16825 hybrid_clock.cc:648] HybridClock initialized: now 1790512745944837 us; error 38 us; skew 500 ppm
Sep 27 12:39:05 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:05.947615 16825 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:05.948602 16825 webserver.cc:460] Webserver started at http://127.0.98.62:37895/ using document root <none> and password file <none>
I20260927 12:39:05.949124 16825 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:05.949186 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:05.953219 16825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:39:05.955421 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:05.956823 16825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:05.956970 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "7b1fa084526d4d4585c562f7b5c938c6"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "1aab24d57442702218edeb97dcb4c1d4"
server_key_iv: "b3aaa840a0fbc9abdd3d0f8ce1c455dd"
server_key_version: "encryptionkey@0"
I20260927 12:39:05.957578 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:05.974311 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:05.978075 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:05.978276 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:05.985603 16825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34847
I20260927 12:39:05.985862 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34847 every 8 connection(s)
I20260927 12:39:05.986698 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:05.989063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16825
I20260927 12:39:05.989509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43577
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34847
--webserver_port=45309
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:39:05.991803 16893 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:39:05.993920 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap starting.
I20260927 12:39:06.011963 16893 log.cc:824] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:06.023258 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:39:06.023726 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: Bootstrap complete.
I20260927 12:39:06.027666 16893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:39:06.028132 16893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1fa084526d4d4585c562f7b5c938c6, State: Initialized, Role: FOLLOWER
I20260927 12:39:06.028667 16893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:39:06.028805 16893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:06.028874 16893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:06.028970 16893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:39:06.030942 16893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:39:06.031276 16893 leader_election.cc:302] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b1fa084526d4d4585c562f7b5c938c6; no voters:
I20260927 12:39:06.031621 16893 leader_election.cc:288] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 12:39:06.031764 16897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:39:06.032027 16897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [term 3 LEADER]: Becoming Leader. State: Replica: 7b1fa084526d4d4585c562f7b5c938c6, State: Running, Role: LEADER
I20260927 12:39:06.032454 16897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } }
I20260927 12:39:06.032856 16893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:06.034024 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7b1fa084526d4d4585c562f7b5c938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } }
I20260927 12:39:06.034152 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:06.034453 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b1fa084526d4d4585c562f7b5c938c6. Latest consensus state: current_term: 3 leader_uuid: "7b1fa084526d4d4585c562f7b5c938c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b1fa084526d4d4585c562f7b5c938c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34847 } } }
I20260927 12:39:06.034538 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:06.037904 16912 catalog_manager.cc:1259] Loaded cluster ID: 17d5a043628c444595efa6ed071c30a4
I20260927 12:39:06.037969 16912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: loading cluster ID for follower catalog manager: success
I20260927 12:39:06.040071 16912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b1fa084526d4d4585c562f7b5c938c6: acquiring CA information for follower catalog manager: success
F20260927 12:39:06.041236 16912 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4c30a29ad google::LogMessage::Fail() at ??:0
@ 0x7fe4c30a588a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4c30a24de google::LogMessage::Flush() at ??:0
@ 0x7fe4c30a3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4c5d6618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe4cda0d989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fe4cd9ea023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fe4cd9ea54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fe4cd9e3652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fe4cd9e29e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fe4cda29541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4c52e5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4c692d6db start_thread at ??:0
@ 0x7fe4c1ba061f clone at ??:0
W20260927 12:39:06.127996 16895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:06.128329 16895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:06.128396 16895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:06.132570 16895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:06.132640 16895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:06.132720 16895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:06.138229 16895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45309
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:06.139307 16895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:06.140367 16895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:06.150497 16933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:06.150497 16934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:06.150694 16895 server_base.cc:1034] running on GCE node
W20260927 12:39:06.150847 16936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:06.151264 16895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:06.151927 16895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:06.153122 16895 hybrid_clock.cc:648] HybridClock initialized: now 1790512746153094 us; error 43 us; skew 500 ppm
Sep 27 12:39:06 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:06.156064 16895 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:39:06.157116 16895 webserver.cc:460] Webserver started at http://127.0.98.1:45309/ using document root <none> and password file <none>
I20260927 12:39:06.157701 16895 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:06.157773 16895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:06.161891 16895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:39:06.164323 16943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.165719 16895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:39:06.165832 16895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "ed8396af3ee64b93b9230c8be15e5b5c"
format_stamp: "Formatted at 2026-09-27 12:38:23 on dist-test-slave-tzgx"
server_key: "066e0cbbbdbc9cc917a86f2dc85a5dfa"
server_key_iv: "ac017d09319f61453c6219256cacc165"
server_key_version: "encryptionkey@0"
I20260927 12:39:06.166437 16895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:06.196836 16895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:06.199975 16895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:06.200186 16895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:06.200825 16895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:06.201941 16950 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 12:39:06.207653 16950 ts_tablet_manager.cc:1852] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 12:39:06.208386 16950 ts_tablet_manager.cc:1905] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:06.214464 16950 ts_tablet_manager.cc:1918] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:06.214637 16950 log.cc:1196] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/wal/wals/f3885a1f1a354a8089663d4039e7793e
I20260927 12:39:06.215101 16950 ts_tablet_manager.cc:1939] T f3885a1f1a354a8089663d4039e7793e P ed8396af3ee64b93b9230c8be15e5b5c: Deleting consensus metadata
I20260927 12:39:06.215508 16895 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 12:39:06.215588 16895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.000s
I20260927 12:39:06.215668 16895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:06.215694 16895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.225912 16895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43577
I20260927 12:39:06.226022 17057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43577 every 8 connection(s)
I20260927 12:39:06.226891 16895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:06.228479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 16895
I20260927 12:39:06.228950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41001
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34847
--webserver_port=41603
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:06.241020 17060 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:06.229117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:39221)
0927 12:39:06.229648 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.229705 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:06.230712 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:06.239888 (+ 9176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
W20260927 12:39:06.241667 17058 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34847 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34847: Client connection negotiation failed: client connection to 127.0.98.62:34847: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:39:06.242437 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.242157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:36055)
0927 12:39:06.242305 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.242375 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 12:39:06.243085 17060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.242876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:38409)
0927 12:39:06.242977 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.243026 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:39:06.243285 17058 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:39:06.342590 17061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:06.342877 17061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:06.342936 17061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:06.347182 17061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:06.347249 17061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:06.347330 17061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:06.353017 17061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41603
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:06.354139 17061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:06.355181 17061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:06.365602 17069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:06.365715 17066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:06.365664 17067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:06.365926 17061 server_base.cc:1034] running on GCE node
I20260927 12:39:06.366396 17061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:06.367097 17061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:06.368268 17061 hybrid_clock.cc:648] HybridClock initialized: now 1790512746368220 us; error 63 us; skew 500 ppm
Sep 27 12:39:06 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:06.371266 17061 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:39:06.372246 17061 webserver.cc:460] Webserver started at http://127.0.98.2:41603/ using document root <none> and password file <none>
I20260927 12:39:06.372787 17061 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:06.372851 17061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:06.377013 17061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:39:06.379377 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.380766 17061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:06.380885 17061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "52290e478d3e427d953202f45c6d7840"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "de4f5f8410329b8ec53af0bcf93a7e29"
server_key_iv: "e1b762e24544c276d665a6fe3b81a699"
server_key_version: "encryptionkey@0"
I20260927 12:39:06.381482 17061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:06.393957 17061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:06.397488 17061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:06.397660 17061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:06.398226 17061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:06.399048 17061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:06.399094 17061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.399133 17061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:06.399153 17061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.408048 17061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41001
I20260927 12:39:06.408118 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41001 every 8 connection(s)
I20260927 12:39:06.409173 17061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:06.413411 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.411645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:53671)
0927 12:39:06.412499 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.412639 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":145,"threads_started":1}
W20260927 12:39:06.413938 17190 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34847 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34847: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
I20260927 12:39:06.414613 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.414337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:59097)
0927 12:39:06.414498 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.414552 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:39:06.415290 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.415058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:43627)
0927 12:39:06.415191 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.415238 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 12:39:06.415520 17190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:39:06.416016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17061
I20260927 12:39:06.416417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38013
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34847
--webserver_port=35327
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:06.528182 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:06.528476 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:06.528522 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:06.532719 17193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:06.532787 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:06.532846 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:06.538780 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35327
--tserver_master_addrs=127.0.98.62:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:06.539960 17193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:06.541050 17193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:06.551275 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:06.551331 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:06.551331 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:06.551558 17193 server_base.cc:1034] running on GCE node
I20260927 12:39:06.552165 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:06.552888 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:06.554088 17193 hybrid_clock.cc:648] HybridClock initialized: now 1790512746554055 us; error 35 us; skew 500 ppm
Sep 27 12:39:06 dist-test-slave-tzgx krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:06.557008 17193 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:39:06.557986 17193 webserver.cc:460] Webserver started at http://127.0.98.3:35327/ using document root <none> and password file <none>
I20260927 12:39:06.558590 17193 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:06.558666 17193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:06.563001 17193 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:39:06.565611 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.567514 17193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:39:06.567672 17193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "a6f90b5bf39b4d6e804ad7dc51ca5194"
format_stamp: "Formatted at 2026-09-27 12:38:24 on dist-test-slave-tzgx"
server_key: "fa412e36b4266aaaf145faa10dc7b100"
server_key_iv: "6df88fd9af5573ad4bd5830a619a2ca4"
server_key_version: "encryptionkey@0"
I20260927 12:39:06.568347 17193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:06.593964 17193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:06.597429 17193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:06.597628 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:06.598242 17193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:06.599085 17193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:06.599133 17193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.599185 17193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:06.599212 17193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:06.608552 17193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38013
I20260927 12:39:06.608573 17321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38013 every 8 connection(s)
I20260927 12:39:06.609576 17193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:06.613423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17193
I20260927 12:39:06.613631 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.611821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:34427)
0927 12:39:06.612337 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.612856 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":182,"threads_started":1}
W20260927 12:39:06.614090 17322 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34847 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34847: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
I20260927 12:39:06.614687 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.614437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:41047)
0927 12:39:06.614588 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.614631 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 12:39:06.615216 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:06.615043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:40189)
0927 12:39:06.615133 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:06.615169 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 12:39:06.615383 17322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:39:07.244609 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:07.244067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:58189)
0927 12:39:07.244435 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:07.244504 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":85,"threads_started":1}
I20260927 12:39:07.416667 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:07.416188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:58607)
0927 12:39:07.416453 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:07.416511 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
I20260927 12:39:07.616348 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:07.615887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:54437)
0927 12:39:07.616185 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:07.616243 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260927 12:39:08.246031 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:08.245545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:34887)
0927 12:39:08.245849 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:08.245918 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260927 12:39:08.418010 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:08.417573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:60059)
0927 12:39:08.417848 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:08.417915 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260927 12:39:08.617638 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:08.617161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:34021)
0927 12:39:08.617471 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:08.617535 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260927 12:39:09.247480 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:09.246962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:40087)
0927 12:39:09.247266 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:09.247329 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
I20260927 12:39:09.419385 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:09.418941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:36591)
0927 12:39:09.419226 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:09.419291 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260927 12:39:09.618911 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:09.618486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:42585)
0927 12:39:09.618749 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:09.618812 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260927 12:39:10.250005 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:10.249521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:39059)
0927 12:39:10.249822 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:10.249884 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260927 12:39:10.420894 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:10.420333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:57707)
0927 12:39:10.420730 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:10.420792 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"threads_started":1}
I20260927 12:39:10.620286 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:10.619714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:49541)
0927 12:39:10.620053 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:10.620181 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":86,"threads_started":1}
I20260927 12:39:11.251538 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:11.250974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:41927)
0927 12:39:11.251290 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:11.251374 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260927 12:39:11.422200 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:11.421748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:52131)
0927 12:39:11.422043 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:11.422105 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260927 12:39:11.621663 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:11.621181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:54945)
0927 12:39:11.621492 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:11.621557 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
I20260927 12:39:12.252892 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:12.252429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:58809)
0927 12:39:12.252738 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:12.252797 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
I20260927 12:39:12.423633 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:12.423150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:57407)
0927 12:39:12.423468 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:12.423534 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":77,"threads_started":1}
I20260927 12:39:12.623019 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:12.622527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:48301)
0927 12:39:12.622847 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:12.622918 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260927 12:39:13.254245 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:13.253742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:45137)
0927 12:39:13.254082 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:13.254146 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":86,"threads_started":1}
I20260927 12:39:13.425109 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:13.424589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:38525)
0927 12:39:13.424946 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:13.425011 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":82,"threads_started":1}
I20260927 12:39:13.624475 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:13.623922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:50323)
0927 12:39:13.624309 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:13.624376 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":78,"threads_started":1}
I20260927 12:39:14.255714 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:14.255188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:59361)
0927 12:39:14.255516 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:14.255596 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":74,"threads_started":1}
I20260927 12:39:14.426563 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:14.426089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:56509)
0927 12:39:14.426353 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:14.426412 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260927 12:39:14.625856 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:14.625443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:58575)
0927 12:39:14.625700 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:14.625759 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260927 12:39:15.257149 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:15.256641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:56707)
0927 12:39:15.256975 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:15.257045 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260927 12:39:15.427892 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:15.427445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:42445)
0927 12:39:15.427729 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:15.427795 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260927 12:39:15.627193 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:15.626768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:58809)
0927 12:39:15.627034 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:15.627097 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260927 12:39:16.258540 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:16.258071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:59385)
0927 12:39:16.258357 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:16.258425 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260927 12:39:16.429257 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:16.428797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:49587)
0927 12:39:16.429095 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:16.429160 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260927 12:39:16.628722 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:16.628229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:42931)
0927 12:39:16.628550 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:16.628621 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":117,"threads_started":1}
I20260927 12:39:17.259831 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:17.259369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:48767)
0927 12:39:17.259666 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:17.259731 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260927 12:39:17.430572 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:17.430104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:49111)
0927 12:39:17.430413 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:17.430470 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260927 12:39:17.630244 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:17.629747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:55095)
0927 12:39:17.630079 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:17.630134 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260927 12:39:18.261358 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:18.260677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:55029)
0927 12:39:18.261072 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:18.261214 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":75,"threads_started":1}
I20260927 12:39:18.432339 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:18.431644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:47977)
0927 12:39:18.432072 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:18.432217 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":112,"threads_started":1}
I20260927 12:39:18.631916 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:18.631252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:43535)
0927 12:39:18.631715 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:18.631808 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":109,"threads_started":1}
I20260927 12:39:19.262742 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:19.262254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:47649)
0927 12:39:19.262569 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:19.262637 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":77,"threads_started":1}
I20260927 12:39:19.433686 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:19.433252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:43397)
0927 12:39:19.433521 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:19.433589 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260927 12:39:19.633399 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:19.632944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:46571)
0927 12:39:19.633242 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:19.633302 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260927 12:39:20.263973 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:20.263536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:60155)
0927 12:39:20.263792 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:20.263874 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260927 12:39:20.435178 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:20.434640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:58551)
0927 12:39:20.434993 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:20.435075 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":99,"threads_started":1}
I20260927 12:39:20.634784 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:20.634302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.3:35129)
0927 12:39:20.634610 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:20.634678 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260927 12:39:21.265415 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:21.264960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.1:45279)
0927 12:39:21.265252 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:21.265314 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260927 12:39:21.436836 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:21.436409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34847 (local address 127.0.98.2:48159)
0927 12:39:21.436674 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:21.436736 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260927 12:39:21.615430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 16895
I20260927 12:39:21.620630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17061
I20260927 12:39:21.625581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17193
2026-09-27T12:39:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59373 ms)
[----------] 30 tests from SecurityITest (248851 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:21 dist-test-slave-tzgx krb5kdc[17372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:21 dist-test-slave-tzgx krb5kdc[17372](info): set up 2 sockets
Sep 27 12:39:21 dist-test-slave-tzgx krb5kdc[17372](info): commencing operation
krb5kdc: starting...
W20260927 12:39:22.603936 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:22 dist-test-slave-tzgx krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:22.728780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:22.839211 17385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:22.839535 17385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:22.839592 17385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:22.843813 17385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:22.843904 17385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:22.843931 17385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:22.843961 17385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:22.843981 17385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:22.849572 17385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:22.850719 17385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:22.851930 17385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:22.861162 17390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:22.861147 17393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:22.861243 17385 server_base.cc:1034] running on GCE node
W20260927 12:39:22.861151 17391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:22.861934 17385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:22.863140 17385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:22.864355 17385 hybrid_clock.cc:648] HybridClock initialized: now 1790512762864334 us; error 36 us; skew 500 ppm
Sep 27 12:39:22 dist-test-slave-tzgx krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:22.867333 17385 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:22.868352 17385 webserver.cc:460] Webserver started at http://127.0.98.62:44097/ using document root <none> and password file <none>
I20260927 12:39:22.868932 17385 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:22.868997 17385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:22.869168 17385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:22.871250 17385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "2893a05dea104e79896e20ad181a8466"
format_stamp: "Formatted at 2026-09-27 12:39:22 on dist-test-slave-tzgx"
server_key: "b192d1d64600229efc7f5dd39731038f"
server_key_iv: "2623eecc3cd45894a096c7721cd7d4b6"
server_key_version: "encryptionkey@0"
I20260927 12:39:22.871774 17385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "2893a05dea104e79896e20ad181a8466"
format_stamp: "Formatted at 2026-09-27 12:39:22 on dist-test-slave-tzgx"
server_key: "b192d1d64600229efc7f5dd39731038f"
server_key_iv: "2623eecc3cd45894a096c7721cd7d4b6"
server_key_version: "encryptionkey@0"
I20260927 12:39:22.875602 17385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:39:22.878356 17400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:22.879802 17385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:22.879941 17385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "2893a05dea104e79896e20ad181a8466"
format_stamp: "Formatted at 2026-09-27 12:39:22 on dist-test-slave-tzgx"
server_key: "b192d1d64600229efc7f5dd39731038f"
server_key_iv: "2623eecc3cd45894a096c7721cd7d4b6"
server_key_version: "encryptionkey@0"
I20260927 12:39:22.880084 17385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:22.904865 17385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:22.908560 17385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:22.908778 17385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:22.915982 17385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39307
I20260927 12:39:22.916018 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39307 every 8 connection(s)
I20260927 12:39:22.917034 17385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:22.919941 17453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:22.925906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17385
I20260927 12:39:22.925952 17453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: Bootstrap starting.
I20260927 12:39:22.926025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:22.926268 392 external_mini_cluster.cc:1428] Setting key 9bb8fbfc6c2a08b4d65577f9bd1b29a5
I20260927 12:39:22.928632 17453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:22.929404 17453 log.cc:824] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:22.931520 17453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: No bootstrap required, opened a new log
Sep 27 12:39:22 dist-test-slave-tzgx krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:22.935952 17453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893a05dea104e79896e20ad181a8466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 12:39:22.936275 17453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:22.936345 17453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2893a05dea104e79896e20ad181a8466, State: Initialized, Role: FOLLOWER
I20260927 12:39:22.936808 17453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893a05dea104e79896e20ad181a8466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 12:39:22.936941 17453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:22.937007 17453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:22.937117 17453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:22.938086 17453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893a05dea104e79896e20ad181a8466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 12:39:22.938473 17453 leader_election.cc:302] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2893a05dea104e79896e20ad181a8466; no voters:
I20260927 12:39:22.938829 17453 leader_election.cc:288] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:22.938961 17458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:22.939177 17458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [term 1 LEADER]: Becoming Leader. State: Replica: 2893a05dea104e79896e20ad181a8466, State: Running, Role: LEADER
I20260927 12:39:22.939543 17458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893a05dea104e79896e20ad181a8466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } }
I20260927 12:39:22.940291 17453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:22.941164 17459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2893a05dea104e79896e20ad181a8466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893a05dea104e79896e20ad181a8466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } } }
I20260927 12:39:22.941300 17459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:22.941606 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:22.927718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.0.98.62:39307)
0927 12:39:22.928195 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:22.928212 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:22.928249 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:39:22.928976 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:22.928997 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:22.929359 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:22.929675 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:22.930443 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:22.931456 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:22.932663 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:22.933169 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:22.933297 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:22.936197 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:22.936203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:22.936218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:22.936251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:22.938060 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:22.938563 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:22.938569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:22.938574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:22.938721 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:22.939035 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:22.939038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:22.939040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:22.939576 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:22.939774 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:22.940239 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 12:39:22.940564 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":143,"threads_started":1}
I20260927 12:39:22.941609 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2893a05dea104e79896e20ad181a8466. Latest consensus state: current_term: 1 leader_uuid: "2893a05dea104e79896e20ad181a8466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2893a05dea104e79896e20ad181a8466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39307 } } }
I20260927 12:39:22.942014 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466 [sys.catalog]: This master's current role is: LEADER
W20260927 12:39:22.945410 17474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:22.945494 17474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:22.945550 17467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:22.946224 17467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:22.952325 17467 catalog_manager.cc:1347] Generated new cluster ID: 4e39f0e90ae14697b63f704990c747d8
I20260927 12:39:22.952425 17467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:22.976559 17467 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:22.977689 17467 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:22.984079 17467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: Generated new TSK 0
I20260927 12:39:22.984885 17467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:39:23.019995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39307
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:23.132751 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:23.133062 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:23.133107 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:23.137890 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:23.137976 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:23.138041 17479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:23.143779 17479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:23.145092 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:23.146243 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:23.156601 17487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:23.156605 17484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:23.156705 17479 server_base.cc:1034] running on GCE node
W20260927 12:39:23.156641 17485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:23.157310 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:23.158079 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:23.159268 17479 hybrid_clock.cc:648] HybridClock initialized: now 1790512763159246 us; error 34 us; skew 500 ppm
Sep 27 12:39:23 dist-test-slave-tzgx krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:23.162590 17479 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:39:23.163865 17479 webserver.cc:460] Webserver started at http://127.0.98.1:33195/ using document root <none> and password file <none>
I20260927 12:39:23.164527 17479 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:23.164595 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:23.164803 17479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:23.167110 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "696de3b2212049839d716ec51fe891f5"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "9ec3383dc801d729e75d53449321b24e"
server_key_iv: "1199d47890e9c519fb0fcad5753e8da3"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.167721 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "696de3b2212049839d716ec51fe891f5"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "9ec3383dc801d729e75d53449321b24e"
server_key_iv: "1199d47890e9c519fb0fcad5753e8da3"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.171840 17479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:39:23.174633 17494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.175988 17479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:23.176146 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "696de3b2212049839d716ec51fe891f5"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "9ec3383dc801d729e75d53449321b24e"
server_key_iv: "1199d47890e9c519fb0fcad5753e8da3"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.176272 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:23.187986 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:23.191304 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:23.191493 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:23.192137 17479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:23.193051 17479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:23.193104 17479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.193228 17479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:23.193311 17479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.202472 17479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35647
I20260927 12:39:23.202527 17607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35647 every 8 connection(s)
I20260927 12:39:23.203437 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:23.206646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17479
I20260927 12:39:23.206784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:39:23.207109 392 external_mini_cluster.cc:1428] Setting key b4e91217e22bfd03cd77796eb90b9864
Sep 27 12:39:23 dist-test-slave-tzgx krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:39:23.219862 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.205609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39491 (local address 127.0.98.62:39307)
0927 12:39:23.205748 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.205751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.206755 (+ 1004us) server_negotiation.cc:407] Connection header received
0927 12:39:23.208368 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.208372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.208417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.208557 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:23.210238 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.211082 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.212446 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.212818 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.212951 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.215931 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:23.215935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:23.215937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:23.215967 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:23.217420 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.218102 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.218105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.218107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.218188 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.218613 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.218616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.218618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.218884 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:23.218993 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.219578 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.219718 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:39:23.220796 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.205947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39307 (local address 127.0.98.1:39491)
0927 12:39:23.206489 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.206595 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.208139 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.208795 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.208806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.209326 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:23.210054 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.210074 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.211225 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.211228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.212255 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.212266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.213019 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.213023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.213205 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.213870 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:23.213889 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:23.215691 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:23.217567 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:23.217574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:23.217594 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:23.217989 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:23.218299 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:23.218302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:23.218303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:23.218473 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:23.219025 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:23.219031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:23.219314 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.219505 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.219803 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:39:23.222251 17608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39307
I20260927 12:39:23.222508 17608 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:23.223059 17608 heartbeater.cc:511] Master 127.0.98.62:39307 requested a full tablet report, sending...
I20260927 12:39:23.224817 17417 ts_manager.cc:194] Registered new tserver with Master: 696de3b2212049839d716ec51fe891f5 (127.0.98.1:35647)
I20260927 12:39:23.226167 17417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:39:23.239301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39307
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:23.354844 17613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:23.355195 17613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:23.355261 17613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:23.360181 17613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:23.360270 17613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:23.360358 17613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:23.366238 17613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:23.367559 17613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:23.368762 17613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:23.379065 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:23.379253 17613 server_base.cc:1034] running on GCE node
W20260927 12:39:23.379112 17621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:23.379163 17618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:23.379915 17613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:23.380688 17613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:23.381889 17613 hybrid_clock.cc:648] HybridClock initialized: now 1790512763381866 us; error 34 us; skew 500 ppm
Sep 27 12:39:23 dist-test-slave-tzgx krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:23.384775 17613 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:39:23.385809 17613 webserver.cc:460] Webserver started at http://127.0.98.2:38469/ using document root <none> and password file <none>
I20260927 12:39:23.386395 17613 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:23.386467 17613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:23.386677 17613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:23.388917 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "00257b265e914be78bc0e1216a0723c1"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "3b2ecdd25281da626549218ea9e84b97"
server_key_iv: "6b9555277602a199274fa885760f7853"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.389492 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "00257b265e914be78bc0e1216a0723c1"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "3b2ecdd25281da626549218ea9e84b97"
server_key_iv: "6b9555277602a199274fa885760f7853"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.393656 17613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:23.396412 17628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.397779 17613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:23.397922 17613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "00257b265e914be78bc0e1216a0723c1"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "3b2ecdd25281da626549218ea9e84b97"
server_key_iv: "6b9555277602a199274fa885760f7853"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.398051 17613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:23.420950 17613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:23.424500 17613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:23.424712 17613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:23.425555 17613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:23.426496 17613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:23.426550 17613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.426608 17613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:23.426638 17613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.435470 17613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40095
I20260927 12:39:23.435489 17741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40095 every 8 connection(s)
I20260927 12:39:23.436429 17613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:23.436806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17613
I20260927 12:39:23.436944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:39:23.437248 392 external_mini_cluster.cc:1428] Setting key 1104e7f878abf0484f630ba483c261bd
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 12:39:23 dist-test-slave-tzgx krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:39:23.451327 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.438428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58571 (local address 127.0.98.62:39307)
0927 12:39:23.438767 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.438772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.439532 (+ 760us) server_negotiation.cc:407] Connection header received
0927 12:39:23.441041 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.441046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.441102 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.441189 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:23.442601 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.443414 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.444675 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.444901 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.444994 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.447541 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:23.447545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:23.447547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:23.447575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:23.449068 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.449680 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.449684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.449686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.449776 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.450123 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.450126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.450127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.450378 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:23.450478 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.451007 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.451155 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260927 12:39:23.452484 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.438854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39307 (local address 127.0.98.2:58571)
0927 12:39:23.439376 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.439423 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.440776 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.441413 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.441423 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.441885 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:23.442399 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.442416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.443563 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.443566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.444544 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.444555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.445051 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.445055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.445226 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.445853 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:23.445870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:23.447340 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:23.449190 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:23.449195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:23.449209 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:23.449552 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:23.449868 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:23.449871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:23.449873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:23.450027 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:23.450489 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:23.450495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:23.450760 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.450942 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.451331 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":149,"threads_started":1}
I20260927 12:39:23.453776 17742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39307
I20260927 12:39:23.454027 17742 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:23.454567 17742 heartbeater.cc:511] Master 127.0.98.62:39307 requested a full tablet report, sending...
I20260927 12:39:23.455861 17417 ts_manager.cc:194] Registered new tserver with Master: 00257b265e914be78bc0e1216a0723c1 (127.0.98.2:40095)
I20260927 12:39:23.456475 17417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:39:23.468891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39307
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:23.580900 17747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:23.581207 17747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:23.581264 17747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:23.586036 17747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:23.586123 17747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:23.586220 17747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:23.592016 17747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:23.593214 17747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:23.594244 17747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:23.604900 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:23.604995 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:23.604995 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:23.605480 17747 server_base.cc:1034] running on GCE node
I20260927 12:39:23.605942 17747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:23.606642 17747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:23.607820 17747 hybrid_clock.cc:648] HybridClock initialized: now 1790512763607785 us; error 45 us; skew 500 ppm
Sep 27 12:39:23 dist-test-slave-tzgx krb5kdc[17372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:23.610709 17747 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:39:23.611661 17747 webserver.cc:460] Webserver started at http://127.0.98.3:37313/ using document root <none> and password file <none>
I20260927 12:39:23.612277 17747 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:23.612349 17747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:23.612555 17747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:23.614630 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "77d83a7892e140e9aa22e83bae98e51b"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "eac33eae97562d18dbc9dcb25ad40bd6"
server_key_iv: "9579f7c160fec0317b872f688bd24f3f"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.615247 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77d83a7892e140e9aa22e83bae98e51b"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "eac33eae97562d18dbc9dcb25ad40bd6"
server_key_iv: "9579f7c160fec0317b872f688bd24f3f"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.619149 17747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 12:39:23.621753 17762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.623131 17747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:39:23.623260 17747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "77d83a7892e140e9aa22e83bae98e51b"
format_stamp: "Formatted at 2026-09-27 12:39:23 on dist-test-slave-tzgx"
server_key: "eac33eae97562d18dbc9dcb25ad40bd6"
server_key_iv: "9579f7c160fec0317b872f688bd24f3f"
server_key_version: "encryptionkey@0"
I20260927 12:39:23.623379 17747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:23.641815 17747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:23.645334 17747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:23.645507 17747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:23.646123 17747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:23.646955 17747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:23.647003 17747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.647043 17747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:23.647065 17747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:23.655710 17747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43855
I20260927 12:39:23.655731 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43855 every 8 connection(s)
I20260927 12:39:23.656656 17747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:23.665863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17747
Sep 27 12:39:23 dist-test-slave-tzgx krb5kdc[17372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:23.665992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:39:23.666293 392 external_mini_cluster.cc:1428] Setting key c0e91484bd7c0732f1e3f69870fe21fc
I20260927 12:39:23.670491 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.658511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39837 (local address 127.0.98.62:39307)
0927 12:39:23.658647 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.658651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.659418 (+ 767us) server_negotiation.cc:407] Connection header received
0927 12:39:23.660716 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.660720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.660778 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.660848 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:23.662014 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.662539 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.663661 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.663885 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.663983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.666733 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:23.666737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:23.666740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:23.666771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:23.668356 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.668916 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.668918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.668920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.669019 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.669393 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.669396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.669397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.669605 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:23.669696 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.670218 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.670330 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:39:23.671269 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.658853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39307 (local address 127.0.98.3:39837)
0927 12:39:23.659271 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.659310 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.660509 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.661028 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.661034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.661391 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:23.661867 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.661881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.662655 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.662658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.663537 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.663547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.664039 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.664043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.664243 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.664822 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:23.664839 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:23.666498 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:23.668488 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:23.668495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:23.668512 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:23.668794 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:23.669143 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:23.669146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:23.669148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:23.669303 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:23.669694 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:23.669699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:23.669970 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.670132 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.670406 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260927 12:39:23.672492 17876 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39307
I20260927 12:39:23.672696 17876 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:23.673153 17876 heartbeater.cc:511] Master 127.0.98.62:39307 requested a full tablet report, sending...
I20260927 12:39:23.674139 17417 ts_manager.cc:194] Registered new tserver with Master: 77d83a7892e140e9aa22e83bae98e51b (127.0.98.3:43855)
I20260927 12:39:23.674683 17417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39837
I20260927 12:39:23.680583 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:39:23.690152 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.682672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.0.98.62:39307)
0927 12:39:23.682835 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.682839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.682976 (+ 137us) server_negotiation.cc:407] Connection header received
0927 12:39:23.683118 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.683121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.683169 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.683235 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:23.683956 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.684539 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.685367 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.685595 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.685683 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.687024 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:23.687028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:23.687031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:23.687058 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:23.688518 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.688913 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.688917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.688919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.689008 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:23.689246 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:23.689249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:23.689251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:23.689483 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:23.689604 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.689825 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.689957 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:39:23.693339 17417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:39:23.696669 17417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:39:23.711737 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.705758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35647 (local address 127.0.0.1:53838)
0927 12:39:23.706246 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.706304 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.706446 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.707280 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.707286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.707309 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.707467 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.707474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.708702 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.708706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.709815 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.709826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.711143 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.711147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.711273 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.711314 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.711441 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.711575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":68,"threads_started":1}
I20260927 12:39:23.711750 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.705911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.0.98.1:35647)
0927 12:39:23.706337 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.706354 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.706379 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:39:23.706736 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.706743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.706966 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.707115 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.707579 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.708588 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.709965 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.710832 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.710938 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.711101 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.711480 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.711587 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":121,"threads_started":1}
I20260927 12:39:23.712172 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.706781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43855 (local address 127.0.0.1:56892)
0927 12:39:23.706981 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.706993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.707063 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.708044 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.708047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.708065 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.708176 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.708181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.709641 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.709644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.710902 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.710913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.711802 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.711806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.711889 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.711925 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.711975 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.712032 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":99,"thread_start_us":82,"threads_started":1}
I20260927 12:39:23.712491 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.706429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40095 (local address 127.0.0.1:50860)
0927 12:39:23.706670 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.706682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.706763 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.707852 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.707855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.707872 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.707969 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.707976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.709646 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.709648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.710752 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.710762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.711845 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.711849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.711967 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.711989 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.712033 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.712129 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":198,"thread_start_us":119,"threads_started":1}
I20260927 12:39:23.712716 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.706610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.98.2:40095)
0927 12:39:23.707424 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.707437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.707461 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:39:23.707533 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.707537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.707714 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.707864 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.708110 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.709464 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.710875 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.711691 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.712241 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.712394 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.712443 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.712538 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":153,"threads_started":1}
I20260927 12:39:23.712826 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.707259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.3:43855)
0927 12:39:23.707640 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.707648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.707664 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:39:23.707720 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.707723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.707867 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.708054 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.708272 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.709493 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.711028 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.711665 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.712346 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.712493 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.712555 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.712681 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":92,"threads_started":1}
I20260927 12:39:23.715531 17676 tablet_service.cc:1467] Processing CreateTablet for tablet f6d90c8009df415b9b220e184fc510f1 (DEFAULT_TABLE table=test-table [id=2a32008ee3a641bc8cd6c8afe1fda9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:23.715561 17542 tablet_service.cc:1467] Processing CreateTablet for tablet f6d90c8009df415b9b220e184fc510f1 (DEFAULT_TABLE table=test-table [id=2a32008ee3a641bc8cd6c8afe1fda9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:23.715561 17810 tablet_service.cc:1467] Processing CreateTablet for tablet f6d90c8009df415b9b220e184fc510f1 (DEFAULT_TABLE table=test-table [id=2a32008ee3a641bc8cd6c8afe1fda9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:23.716634 17676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d90c8009df415b9b220e184fc510f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:23.716645 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d90c8009df415b9b220e184fc510f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:23.716645 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d90c8009df415b9b220e184fc510f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:23.724033 17893 tablet_bootstrap.cc:492] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Bootstrap starting.
I20260927 12:39:23.724464 17895 tablet_bootstrap.cc:492] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Bootstrap starting.
I20260927 12:39:23.726142 17895 tablet_bootstrap.cc:654] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:23.726142 17893 tablet_bootstrap.cc:654] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:23.727226 17893 log.cc:824] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:23.727234 17895 log.cc:824] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:23.727983 17894 tablet_bootstrap.cc:492] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: Bootstrap starting.
I20260927 12:39:23.729410 17893 tablet_bootstrap.cc:492] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: No bootstrap required, opened a new log
I20260927 12:39:23.729620 17893 ts_tablet_manager.cc:1397] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:39:23.730283 17894 tablet_bootstrap.cc:654] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:23.730996 17894 log.cc:824] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:23.733049 17894 tablet_bootstrap.cc:492] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: No bootstrap required, opened a new log
I20260927 12:39:23.732892 17893 raft_consensus.cc:348] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.733278 17894 ts_tablet_manager.cc:1397] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:39:23.733266 17893 raft_consensus.cc:374] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:23.733341 17893 raft_consensus.cc:729] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00257b265e914be78bc0e1216a0723c1, State: Initialized, Role: FOLLOWER
I20260927 12:39:23.733517 17895 tablet_bootstrap.cc:492] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: No bootstrap required, opened a new log
I20260927 12:39:23.733719 17895 ts_tablet_manager.cc:1397] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 12:39:23.733866 17893 consensus_queue.cc:260] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.734644 17742 heartbeater.cc:503] Master 127.0.98.62:39307 was elected leader, sending a full tablet report...
I20260927 12:39:23.734937 17893 ts_tablet_manager.cc:1428] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 12:39:23.737471 17895 raft_consensus.cc:348] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.737733 17895 raft_consensus.cc:374] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:23.737818 17895 raft_consensus.cc:729] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d83a7892e140e9aa22e83bae98e51b, State: Initialized, Role: FOLLOWER
I20260927 12:39:23.738052 17894 raft_consensus.cc:348] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.738274 17895 consensus_queue.cc:260] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.738370 17894 raft_consensus.cc:374] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:23.738440 17894 raft_consensus.cc:729] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696de3b2212049839d716ec51fe891f5, State: Initialized, Role: FOLLOWER
I20260927 12:39:23.738963 17876 heartbeater.cc:503] Master 127.0.98.62:39307 was elected leader, sending a full tablet report...
I20260927 12:39:23.738957 17894 consensus_queue.cc:260] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.739666 17608 heartbeater.cc:503] Master 127.0.98.62:39307 was elected leader, sending a full tablet report...
I20260927 12:39:23.740170 17894 ts_tablet_manager.cc:1428] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:23.740597 17895 ts_tablet_manager.cc:1428] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
W20260927 12:39:23.907657 17877 tablet.cc:2369] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:23.916663 17901 raft_consensus.cc:482] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:23.916894 17901 raft_consensus.cc:504] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.918480 17901 leader_election.cc:288] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00257b265e914be78bc0e1216a0723c1 (127.0.98.2:40095), 77d83a7892e140e9aa22e83bae98e51b (127.0.98.3:43855)
I20260927 12:39:23.923936 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.918884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49831 (local address 127.0.98.2:40095)
0927 12:39:23.919041 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.919046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.919303 (+ 257us) server_negotiation.cc:407] Connection header received
0927 12:39:23.919544 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.919548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.919599 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.919649 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.920080 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.920870 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.922476 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.923101 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.923191 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.923261 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.923697 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.923792 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:39:23.924664 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.918768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40095 (local address 127.0.98.1:49831)
0927 12:39:23.919215 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.919229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.919408 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.919797 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.919800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.919817 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.919947 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.919954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.921029 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.921032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.922304 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.922318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.923373 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.923377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.923543 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.923568 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.924401 (+ 833us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.924499 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":1,"thread_start_us":171,"threads_started":1}
I20260927 12:39:23.925944 17696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d90c8009df415b9b220e184fc510f1" candidate_uuid: "696de3b2212049839d716ec51fe891f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00257b265e914be78bc0e1216a0723c1" is_pre_election: true
I20260927 12:39:23.926317 17696 raft_consensus.cc:2454] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696de3b2212049839d716ec51fe891f5 in term 0.
I20260927 12:39:23.926843 17900 raft_consensus.cc:482] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:23.927067 17496 leader_election.cc:302] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00257b265e914be78bc0e1216a0723c1, 696de3b2212049839d716ec51fe891f5; no voters:
I20260927 12:39:23.927071 17900 raft_consensus.cc:504] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.927459 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.919559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41455 (local address 127.0.98.3:43855)
0927 12:39:23.919727 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.919756 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.921046 (+ 1290us) server_negotiation.cc:407] Connection header received
0927 12:39:23.921220 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.921223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.921269 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.921313 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.921714 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.922431 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.924771 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.926625 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.926797 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.926901 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.927263 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.927361 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:39:23.927542 17901 raft_consensus.cc:2781] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:39:23.927661 17901 raft_consensus.cc:482] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:39:23.927702 17901 raft_consensus.cc:3037] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:23.927857 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.919880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43855 (local address 127.0.98.1:41455)
0927 12:39:23.920930 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.920947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.921114 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.921457 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.921460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.921476 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.921584 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.921590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.922589 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.922592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.924564 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.924579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.926996 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.927000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.927111 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.927131 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.927591 (+ 460us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.927724 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":376,"thread_start_us":126,"threads_started":1}
I20260927 12:39:23.928464 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d90c8009df415b9b220e184fc510f1" candidate_uuid: "696de3b2212049839d716ec51fe891f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d83a7892e140e9aa22e83bae98e51b" is_pre_election: true
I20260927 12:39:23.928670 17830 raft_consensus.cc:2454] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696de3b2212049839d716ec51fe891f5 in term 0.
I20260927 12:39:23.929298 17901 raft_consensus.cc:504] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.929701 17901 leader_election.cc:288] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [CANDIDATE]: Term 1 election: Requested vote from peers 00257b265e914be78bc0e1216a0723c1 (127.0.98.2:40095), 77d83a7892e140e9aa22e83bae98e51b (127.0.98.3:43855)
I20260927 12:39:23.929931 17900 leader_election.cc:288] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 696de3b2212049839d716ec51fe891f5 (127.0.98.1:35647), 00257b265e914be78bc0e1216a0723c1 (127.0.98.2:40095)
I20260927 12:39:23.930990 17696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d90c8009df415b9b220e184fc510f1" candidate_uuid: "696de3b2212049839d716ec51fe891f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00257b265e914be78bc0e1216a0723c1"
I20260927 12:39:23.931092 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d90c8009df415b9b220e184fc510f1" candidate_uuid: "696de3b2212049839d716ec51fe891f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d83a7892e140e9aa22e83bae98e51b"
I20260927 12:39:23.931143 17696 raft_consensus.cc:3037] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:23.931200 17830 raft_consensus.cc:3037] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:23.932585 17830 raft_consensus.cc:2454] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696de3b2212049839d716ec51fe891f5 in term 1.
I20260927 12:39:23.932658 17696 raft_consensus.cc:2454] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696de3b2212049839d716ec51fe891f5 in term 1.
I20260927 12:39:23.933104 17496 leader_election.cc:302] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696de3b2212049839d716ec51fe891f5, 77d83a7892e140e9aa22e83bae98e51b; no voters:
I20260927 12:39:23.933274 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.929067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57709 (local address 127.0.98.1:35647)
0927 12:39:23.929172 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.929176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.929189 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:39:23.929233 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.929235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.929277 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.929342 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.929667 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.930430 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.931960 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.932588 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.932681 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.932746 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.933078 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.933155 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:39:23.933559 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.928730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35647 (local address 127.0.98.3:57709)
0927 12:39:23.929017 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.929035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.929129 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.929440 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.929443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.929463 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.929564 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.929570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.930555 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.930558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.931805 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.931816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.932801 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.932805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.932917 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.932939 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.933002 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.933454 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":349}
I20260927 12:39:23.933559 17901 raft_consensus.cc:2781] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:23.933763 17901 raft_consensus.cc:686] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [term 1 LEADER]: Becoming Leader. State: Replica: 696de3b2212049839d716ec51fe891f5, State: Running, Role: LEADER
I20260927 12:39:23.934172 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d90c8009df415b9b220e184fc510f1" candidate_uuid: "77d83a7892e140e9aa22e83bae98e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696de3b2212049839d716ec51fe891f5" is_pre_election: true
I20260927 12:39:23.934207 17901 consensus_queue.cc:237] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } }
I20260927 12:39:23.936673 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.930182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59419 (local address 127.0.98.2:40095)
0927 12:39:23.930314 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.930319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.930333 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:39:23.930385 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.930388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.930439 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.930513 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:23.933299 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.934021 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.935489 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.936073 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.936212 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.936278 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.936476 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.936555 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:39:23.936682 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.929549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40095 (local address 127.0.98.3:59419)
0927 12:39:23.930117 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.930132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:23.930227 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:23.930628 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:23.930631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:23.930650 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:23.933160 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.933168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.934160 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.934163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.935307 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:23.935317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.936239 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.936243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:23.936341 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.936367 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:39:23.936451 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 12:39:23.936530 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":2360,"thread_start_us":153,"threads_started":1}
I20260927 12:39:23.937193 17696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d90c8009df415b9b220e184fc510f1" candidate_uuid: "77d83a7892e140e9aa22e83bae98e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00257b265e914be78bc0e1216a0723c1" is_pre_election: true
W20260927 12:39:23.937515 17743 tablet.cc:2369] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:23.937431 17416 catalog_manager.cc:5495] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696de3b2212049839d716ec51fe891f5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "696de3b2212049839d716ec51fe891f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696de3b2212049839d716ec51fe891f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } health_report { overall_health: UNKNOWN } } }
I20260927 12:39:23.938004 17696 raft_consensus.cc:2379] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77d83a7892e140e9aa22e83bae98e51b in current term 1: Already voted for candidate 696de3b2212049839d716ec51fe891f5 in this term.
I20260927 12:39:23.938476 17764 leader_election.cc:302] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77d83a7892e140e9aa22e83bae98e51b; no voters: 00257b265e914be78bc0e1216a0723c1, 696de3b2212049839d716ec51fe891f5
I20260927 12:39:23.938730 17900 raft_consensus.cc:2726] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 12:39:23.950276 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.946442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.0.98.1:35647)
0927 12:39:23.946578 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:23.946582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:23.946596 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:39:23.946686 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:23.946688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:23.946742 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:23.946834 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:39:23.947095 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.947644 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.948704 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:23.948945 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:23.949054 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:23.949383 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:39:23.949861 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:39:23.949977 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:23.950062 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 12:39:23.950129 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 12:39:23.954973 17609 tablet.cc:2369] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:23.956671 17696 raft_consensus.cc:1261] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 1 FOLLOWER]: Refusing update from remote peer 696de3b2212049839d716ec51fe891f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:23.956724 17830 raft_consensus.cc:1261] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 1 FOLLOWER]: Refusing update from remote peer 696de3b2212049839d716ec51fe891f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:23.957343 17905 consensus_queue.cc:1035] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00257b265e914be78bc0e1216a0723c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:23.957571 17901 consensus_queue.cc:1035] T f6d90c8009df415b9b220e184fc510f1 P 696de3b2212049839d716ec51fe891f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77d83a7892e140e9aa22e83bae98e51b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:23.968899 17910 mvcc.cc:205] Tried to move back new op lower bound from 7333940281158365184 to 7333940281077653504. Current Snapshot: MvccSnapshot[applied={T|T < 7333940281158365184}]
I20260927 12:39:23.977021 17416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:23.977198 17416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:23.979939 17416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2893a05dea104e79896e20ad181a8466: Sending DeleteTablet for 3 replicas of tablet f6d90c8009df415b9b220e184fc510f1
I20260927 12:39:23.981357 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d90c8009df415b9b220e184fc510f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53838
I20260927 12:39:23.981357 17676 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d90c8009df415b9b220e184fc510f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50860
I20260927 12:39:23.982251 17917 tablet_replica.cc:330] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: stopping tablet replica
I20260927 12:39:23.982247 17810 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d90c8009df415b9b220e184fc510f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56892
I20260927 12:39:23.982607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17479
I20260927 12:39:23.982751 17917 raft_consensus.cc:2229] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:23.983111 17917 raft_consensus.cc:2258] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:23.984884 17919 tablet_replica.cc:330] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: stopping tablet replica
I20260927 12:39:23.985234 17919 raft_consensus.cc:2229] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:23.985601 17919 raft_consensus.cc:2258] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:23.985837 17917 ts_tablet_manager.cc:1905] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:23.987303 17919 ts_tablet_manager.cc:1905] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:23.989270 17917 ts_tablet_manager.cc:1918] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:23.989387 17917 log.cc:1196] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/f6d90c8009df415b9b220e184fc510f1
I20260927 12:39:23.989687 17917 ts_tablet_manager.cc:1939] T f6d90c8009df415b9b220e184fc510f1 P 00257b265e914be78bc0e1216a0723c1: Deleting consensus metadata
I20260927 12:39:23.991029 17919 ts_tablet_manager.cc:1918] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:23.991067 17402 catalog_manager.cc:4841] TS 00257b265e914be78bc0e1216a0723c1 (127.0.98.2:40095): tablet f6d90c8009df415b9b220e184fc510f1 (table test-table [id=2a32008ee3a641bc8cd6c8afe1fda9b4]) successfully deleted
I20260927 12:39:23.991178 17919 log.cc:1196] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/f6d90c8009df415b9b220e184fc510f1
W20260927 12:39:23.991370 17402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35647 (error 108)
I20260927 12:39:23.991542 17919 ts_tablet_manager.cc:1939] T f6d90c8009df415b9b220e184fc510f1 P 77d83a7892e140e9aa22e83bae98e51b: Deleting consensus metadata
I20260927 12:39:23.991580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17613
I20260927 12:39:23.992632 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:23.992286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35647 (local address 127.0.0.1:53852)
0927 12:39:23.992431 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:23.992521 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:39:23.992823 17402 catalog_manager.cc:4584] TS 696de3b2212049839d716ec51fe891f5 (127.0.98.1:35647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6d90c8009df415b9b220e184fc510f1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35647: connect: Connection refused (error 111)
I20260927 12:39:23.993680 17402 catalog_manager.cc:4841] TS 77d83a7892e140e9aa22e83bae98e51b (127.0.98.3:43855): tablet f6d90c8009df415b9b220e184fc510f1 (table test-table [id=2a32008ee3a641bc8cd6c8afe1fda9b4]) successfully deleted
I20260927 12:39:23.997087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17747
I20260927 12:39:24.002365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17385
2026-09-27T12:39:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2376 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T12:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:24Z Disabled control of system clock
I20260927 12:39:24.032433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44099
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:39:24.149667 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:24.149973 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:24.150009 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:24.154242 17924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:24.154326 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:24.154341 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:24.154362 17924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:24.154377 17924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:24.159855 17924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:24.161084 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:24.162528 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:24.172220 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:24.172439 17924 server_base.cc:1034] running on GCE node
W20260927 12:39:24.172220 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:24.172258 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:24.173135 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:24.174355 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:24.175554 17924 hybrid_clock.cc:648] HybridClock initialized: now 1790512764175525 us; error 43 us; skew 500 ppm
I20260927 12:39:24.177757 17924 webserver.cc:460] Webserver started at http://127.0.98.62:39815/ using document root <none> and password file <none>
I20260927 12:39:24.178369 17924 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:24.178447 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:24.178673 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:24.181067 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "1e3b773058e74a128cc4df23a8f89752"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "5aa565e030dbadcfba40d0f6c0b45f90"
server_key_iv: "1040cb55ae99360b2dd1adb70c55ade4"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.181671 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e3b773058e74a128cc4df23a8f89752"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "5aa565e030dbadcfba40d0f6c0b45f90"
server_key_iv: "1040cb55ae99360b2dd1adb70c55ade4"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.185691 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:24.188289 17938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.189721 17924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:24.189868 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "1e3b773058e74a128cc4df23a8f89752"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "5aa565e030dbadcfba40d0f6c0b45f90"
server_key_iv: "1040cb55ae99360b2dd1adb70c55ade4"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.189990 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:24.207619 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:24.208732 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:24.208966 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:24.215924 17924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44099
I20260927 12:39:24.215939 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44099 every 8 connection(s)
I20260927 12:39:24.216946 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:24.219362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 17924
I20260927 12:39:24.219497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:24.219786 392 external_mini_cluster.cc:1428] Setting key 708f4fca1af187e5906afadcea9e75ba
I20260927 12:39:24.220619 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:24.227027 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752: Bootstrap starting.
I20260927 12:39:24.229305 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.221329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.0.98.62:44099)
0927 12:39:24.222002 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.222017 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.222077 (+ 60us) server_negotiation.cc:407] Connection header received
0927 12:39:24.222748 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.222780 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.223219 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.223527 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:24.223956 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.225014 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.225939 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.226453 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.226611 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.227516 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:24.227521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:24.227539 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:24.227942 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:24.228001 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:24.228020 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:24.228140 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.228276 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.228549 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260927 12:39:24.229513 17991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:24.230293 17991 log.cc:824] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:24.232599 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752: No bootstrap required, opened a new log
I20260927 12:39:24.236735 17991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3b773058e74a128cc4df23a8f89752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44099 } }
I20260927 12:39:24.237073 17991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:24.237156 17991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3b773058e74a128cc4df23a8f89752, State: Initialized, Role: FOLLOWER
I20260927 12:39:24.237633 17991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3b773058e74a128cc4df23a8f89752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44099 } }
I20260927 12:39:24.237775 17991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:24.237824 17991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:24.237928 17991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:24.238979 17991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3b773058e74a128cc4df23a8f89752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44099 } }
I20260927 12:39:24.239367 17991 leader_election.cc:302] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e3b773058e74a128cc4df23a8f89752; no voters:
I20260927 12:39:24.239722 17991 leader_election.cc:288] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:24.239846 17996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:24.240078 17996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [term 1 LEADER]: Becoming Leader. State: Replica: 1e3b773058e74a128cc4df23a8f89752, State: Running, Role: LEADER
I20260927 12:39:24.240517 17996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3b773058e74a128cc4df23a8f89752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44099 } }
I20260927 12:39:24.241071 17991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:24.242089 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e3b773058e74a128cc4df23a8f89752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3b773058e74a128cc4df23a8f89752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44099 } } }
I20260927 12:39:24.242230 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:24.242574 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e3b773058e74a128cc4df23a8f89752. Latest consensus state: current_term: 1 leader_uuid: "1e3b773058e74a128cc4df23a8f89752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3b773058e74a128cc4df23a8f89752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44099 } } }
I20260927 12:39:24.242759 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:24.242650 18005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:24.246675 18005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:24.253839 18005 catalog_manager.cc:1347] Generated new cluster ID: bd4a6a5ec2c94128a03f84cb9d891632
I20260927 12:39:24.253925 18005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:24.283138 18005 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:24.284240 18005 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:24.291327 18005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752: Generated new TSK 0
I20260927 12:39:24.292136 18005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:24.304239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44099
--builtin_ntp_servers=127.0.98.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:39:24.428772 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:24.429072 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:24.429112 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:24.433382 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:24.433459 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:24.433517 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:24.438607 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:24.439776 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:24.441257 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:24.452028 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:24.452104 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:24.452224 18015 server_base.cc:1034] running on GCE node
W20260927 12:39:24.452082 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:24.452761 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:24.453485 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:24.454687 18015 hybrid_clock.cc:648] HybridClock initialized: now 1790512764454665 us; error 34 us; skew 500 ppm
I20260927 12:39:24.456908 18015 webserver.cc:460] Webserver started at http://127.0.98.1:44227/ using document root <none> and password file <none>
I20260927 12:39:24.457487 18015 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:24.457552 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:24.457854 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:24.460294 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "96e16936c78e431f897a79e8117d69b1"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "0a3b9bc9cd5c4de09715a33fa287008c"
server_key_iv: "0a96c9a85ea9d6ff0518c08af18d1b31"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.461102 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96e16936c78e431f897a79e8117d69b1"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "0a3b9bc9cd5c4de09715a33fa287008c"
server_key_iv: "0a96c9a85ea9d6ff0518c08af18d1b31"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.465286 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 12:39:24.468212 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.469657 18015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:24.469776 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "96e16936c78e431f897a79e8117d69b1"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "0a3b9bc9cd5c4de09715a33fa287008c"
server_key_iv: "0a96c9a85ea9d6ff0518c08af18d1b31"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.469914 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:24.485596 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:24.486337 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:24.486533 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:24.487151 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:24.488191 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:24.488242 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.488298 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:24.488327 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.497164 18015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40391
I20260927 12:39:24.497195 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40391 every 8 connection(s)
I20260927 12:39:24.498088 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:24.501266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18015
I20260927 12:39:24.501389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:39:24.501643 392 external_mini_cluster.cc:1428] Setting key 2011b1e3e77667cabd3f891588ad2aa6
I20260927 12:39:24.503985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44099
--builtin_ntp_servers=127.0.98.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:39:24.510635 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.499945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44187 (local address 127.0.98.62:44099)
0927 12:39:24.500074 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.500077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.501412 (+ 1335us) server_negotiation.cc:407] Connection header received
0927 12:39:24.503021 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.503025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.503080 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.503124 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:24.504651 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.505339 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.507861 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.508215 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.508361 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.509978 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:24.509983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:24.509985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:24.510009 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:24.510040 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:24.510052 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:24.510127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.510362 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.510491 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:39:24.511885 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.500417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44099 (local address 127.0.98.1:44187)
0927 12:39:24.501119 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:24.501257 (+ 138us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:24.502747 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:24.503709 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:24.503717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:24.503825 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:24.504476 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.504494 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.505515 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.505519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.507603 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.507621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.508456 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.508462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.508749 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.509521 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:24.509523 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:24.509596 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:24.509708 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:24.510152 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:24.510162 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:24.510168 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:39:24.510325 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 12:39:24.510719 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":108,"threads_started":1}
I20260927 12:39:24.513456 18143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44099
I20260927 12:39:24.513731 18143 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:24.514284 18143 heartbeater.cc:511] Master 127.0.98.62:44099 requested a full tablet report, sending...
I20260927 12:39:24.515849 17955 ts_manager.cc:194] Registered new tserver with Master: 96e16936c78e431f897a79e8117d69b1 (127.0.98.1:40391)
I20260927 12:39:24.517052 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44187
W20260927 12:39:24.624756 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:24.625043 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:24.625082 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:24.629272 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:24.629357 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:24.629417 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:24.634666 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:24.636118 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:24.637603 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:24.647898 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:24.647891 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:24.648293 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:24.648434 18146 server_base.cc:1034] running on GCE node
I20260927 12:39:24.648862 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:24.649727 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:24.650997 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790512764650892 us; error 103 us; skew 500 ppm
I20260927 12:39:24.653730 18146 webserver.cc:460] Webserver started at http://127.0.98.2:45815/ using document root <none> and password file <none>
I20260927 12:39:24.654644 18146 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:24.654767 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:24.654999 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:24.657272 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f3564356410483ba6fc572d1706a679"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "c7931d3855138f0ff5ffe32f151601b8"
server_key_iv: "ad93449da8745e6eb2e064e99cd0f4ca"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.657855 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f3564356410483ba6fc572d1706a679"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "c7931d3855138f0ff5ffe32f151601b8"
server_key_iv: "ad93449da8745e6eb2e064e99cd0f4ca"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.661679 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:24.664378 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.665720 18146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:39:24.665856 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "1f3564356410483ba6fc572d1706a679"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "c7931d3855138f0ff5ffe32f151601b8"
server_key_iv: "ad93449da8745e6eb2e064e99cd0f4ca"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.665970 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:24.701968 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:24.702693 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:24.702865 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:24.703519 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:24.704551 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:24.704674 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.704779 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:24.704808 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.713600 18146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46849
I20260927 12:39:24.713649 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46849 every 8 connection(s)
I20260927 12:39:24.714509 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:24.722601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18146
I20260927 12:39:24.722755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:39:24.723078 392 external_mini_cluster.cc:1428] Setting key edb937127f39a525dfd5c9053f3c2b92
I20260927 12:39:24.725816 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.716363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60985 (local address 127.0.98.62:44099)
0927 12:39:24.716533 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.716537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.717598 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 12:39:24.719424 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.719429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.719515 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.719625 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:24.720985 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.721558 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.723398 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.723643 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.723742 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.725216 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:24.725220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:24.725223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:24.725238 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:24.725264 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:24.725275 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:24.725345 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.725570 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.725650 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:39:24.727187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44099
--builtin_ntp_servers=127.0.98.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:39:24.727533 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.716754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44099 (local address 127.0.98.2:60985)
0927 12:39:24.717432 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:24.717474 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:24.719047 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:24.719858 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:24.719869 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:24.720132 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:24.720784 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.720801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.721720 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.721724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.723238 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.723255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.723817 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.723822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.724054 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.724836 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:24.724838 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:24.724903 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:24.725009 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:24.725363 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:24.725373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:24.725380 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:39:24.726038 (+ 658us) client_negotiation.cc:240] Negotiation successful
0927 12:39:24.726419 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260927 12:39:24.729337 18274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44099
I20260927 12:39:24.729597 18274 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:24.730161 18274 heartbeater.cc:511] Master 127.0.98.62:44099 requested a full tablet report, sending...
I20260927 12:39:24.731360 17955 ts_manager.cc:194] Registered new tserver with Master: 1f3564356410483ba6fc572d1706a679 (127.0.98.2:46849)
I20260927 12:39:24.731977 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60985
W20260927 12:39:24.839986 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:24.840317 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:24.840369 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:24.845142 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:24.845227 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:24.845329 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:24.853209 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:24.854743 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:24.856236 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:24.866252 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:24.866529 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:24.866547 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:24.866689 18277 server_base.cc:1034] running on GCE node
I20260927 12:39:24.867193 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:24.867893 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:24.869079 18277 hybrid_clock.cc:648] HybridClock initialized: now 1790512764869060 us; error 42 us; skew 500 ppm
I20260927 12:39:24.871179 18277 webserver.cc:460] Webserver started at http://127.0.98.3:39385/ using document root <none> and password file <none>
I20260927 12:39:24.871774 18277 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:24.871845 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:24.872052 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:24.874290 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "17c8ccedf4db4c2cb665df4734c33e13"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "55901ce87b7a6ce1bec17b80eba4ced1"
server_key_iv: "22425fe8807f6e52cc5f4ab1a3f8c17a"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.874877 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "17c8ccedf4db4c2cb665df4734c33e13"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "55901ce87b7a6ce1bec17b80eba4ced1"
server_key_iv: "22425fe8807f6e52cc5f4ab1a3f8c17a"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.878942 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:39:24.881783 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.883232 18277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:39:24.883348 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "17c8ccedf4db4c2cb665df4734c33e13"
format_stamp: "Formatted at 2026-09-27 12:39:24 on dist-test-slave-tzgx"
server_key: "55901ce87b7a6ce1bec17b80eba4ced1"
server_key_iv: "22425fe8807f6e52cc5f4ab1a3f8c17a"
server_key_version: "encryptionkey@0"
I20260927 12:39:24.883490 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:24.913051 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:24.913790 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:24.913969 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:24.914587 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:24.915606 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:24.915652 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.915695 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:24.915723 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:24.924535 18277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40463
I20260927 12:39:24.924578 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40463 every 8 connection(s)
I20260927 12:39:24.925541 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:24.934988 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.927478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44617 (local address 127.0.98.62:44099)
0927 12:39:24.927641 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.927645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.928741 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 12:39:24.930076 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.930079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.930130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.930206 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:24.931166 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.931714 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.932965 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.933198 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.933279 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.934440 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:24.934444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:24.934446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:24.934461 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:24.934483 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:24.934492 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:24.934584 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.934798 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.934859 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:39:24.935757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18277
I20260927 12:39:24.935843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:39:24.935953 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.927853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44099 (local address 127.0.98.3:44617)
0927 12:39:24.928590 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:24.928632 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:24.929882 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:24.930437 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:24.930445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:24.930531 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:24.931034 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.931047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.931863 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.931866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.932819 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.932830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.933368 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.933372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.933537 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.934153 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:24.934156 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:24.934199 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:24.934272 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:24.934617 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:24.934628 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:24.934633 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:39:24.934759 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 12:39:24.935022 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":114,"threads_started":1}
I20260927 12:39:24.936122 392 external_mini_cluster.cc:1428] Setting key 7fba36c2515046cb94eb51aac18ee4fb
I20260927 12:39:24.937635 18405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44099
I20260927 12:39:24.937911 18405 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:24.938464 18405 heartbeater.cc:511] Master 127.0.98.62:44099 requested a full tablet report, sending...
I20260927 12:39:24.939628 17954 ts_manager.cc:194] Registered new tserver with Master: 17c8ccedf4db4c2cb665df4734c33e13 (127.0.98.3:40463)
I20260927 12:39:24.940356 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44617
I20260927 12:39:24.952310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:39:24.958083 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.954254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.0.98.62:44099)
0927 12:39:24.954420 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.954425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.954571 (+ 146us) server_negotiation.cc:407] Connection header received
0927 12:39:24.954704 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.954707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.954754 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.954822 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:24.955132 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.955654 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.956561 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.956778 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.956876 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.957664 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:24.957667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:24.957669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:24.957682 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:24.957702 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:24.957710 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:24.957770 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.957872 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.957933 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:39:24.961272 17954 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:39:24.964520 17954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:39:24.979200 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.973874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.0.98.2:46849)
0927 12:39:24.974655 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.974667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.974688 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:39:24.974758 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.974764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.974940 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.975107 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:24.975356 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.976594 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.977984 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.978512 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.978610 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.978814 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.978923 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.979037 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":119,"threads_started":1}
I20260927 12:39:24.979197 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.973454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46849 (local address 127.0.0.1:35472)
0927 12:39:24.974283 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:24.974296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:24.974387 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:24.975076 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:24.975079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:24.975095 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:24.975201 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.975207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.976732 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.976735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.977872 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.977879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.978632 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.978635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.978742 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.978779 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:39:24.978889 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:39:24.979037 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":44,"thread_start_us":95,"threads_started":1}
I20260927 12:39:24.979434 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.973343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.98.1:40391)
0927 12:39:24.973704 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.973712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.974080 (+ 368us) server_negotiation.cc:407] Connection header received
0927 12:39:24.974290 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.974295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.974758 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.975270 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:24.975402 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.976221 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.977702 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.978527 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.978628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.979110 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.979157 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.979249 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260927 12:39:24.979739 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.973205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40391 (local address 127.0.0.1:44914)
0927 12:39:24.973937 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:24.974001 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:24.974130 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:24.974925 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:24.974931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:24.974954 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:24.975132 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.975140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.976315 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.976318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.977583 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.977594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.978734 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.978737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.978826 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.978851 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:39:24.978904 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 12:39:24.979611 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":588,"thread_start_us":78,"threads_started":1}
I20260927 12:39:24.979895 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.973903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40463 (local address 127.0.0.1:47850)
0927 12:39:24.974782 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:24.974792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:24.974877 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:24.975338 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:24.975341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:24.975357 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:24.975442 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.975448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.976748 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.976751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.977896 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:24.977906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.979136 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.979139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:24.979222 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.979243 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:39:24.979650 (+ 407us) client_negotiation.cc:240] Negotiation successful
0927 12:39:24.979730 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":58,"thread_start_us":85,"threads_started":1}
I20260927 12:39:24.979926 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:24.974336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.98.3:40463)
0927 12:39:24.974657 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:24.974669 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:24.974882 (+ 213us) server_negotiation.cc:407] Connection header received
0927 12:39:24.974983 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:24.974987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:24.975211 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:24.975357 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:24.975534 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.976642 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.978038 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:24.978796 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:24.978886 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:24.979077 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:24.979683 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 12:39:24.979780 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":156,"threads_started":1}
I20260927 12:39:24.982553 18208 tablet_service.cc:1467] Processing CreateTablet for tablet fff2ed9797a44545a6e9ecbc0c4d6218 (DEFAULT_TABLE table=test-table [id=fc214e82bd014b4ca63a2e5446b73765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:24.982590 18077 tablet_service.cc:1467] Processing CreateTablet for tablet fff2ed9797a44545a6e9ecbc0c4d6218 (DEFAULT_TABLE table=test-table [id=fc214e82bd014b4ca63a2e5446b73765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:24.982590 18339 tablet_service.cc:1467] Processing CreateTablet for tablet fff2ed9797a44545a6e9ecbc0c4d6218 (DEFAULT_TABLE table=test-table [id=fc214e82bd014b4ca63a2e5446b73765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:24.983549 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2ed9797a44545a6e9ecbc0c4d6218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:24.983567 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2ed9797a44545a6e9ecbc0c4d6218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:24.983567 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2ed9797a44545a6e9ecbc0c4d6218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:24.991128 18423 tablet_bootstrap.cc:492] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1: Bootstrap starting.
I20260927 12:39:24.992432 18422 tablet_bootstrap.cc:492] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Bootstrap starting.
I20260927 12:39:24.993239 18423 tablet_bootstrap.cc:654] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:24.994001 18423 log.cc:824] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:24.994202 18422 tablet_bootstrap.cc:654] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:24.994689 18424 tablet_bootstrap.cc:492] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Bootstrap starting.
I20260927 12:39:24.994892 18422 log.cc:824] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:24.996876 18424 tablet_bootstrap.cc:654] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:24.997074 18422 tablet_bootstrap.cc:492] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: No bootstrap required, opened a new log
I20260927 12:39:24.997263 18422 ts_tablet_manager.cc:1397] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:39:24.997706 18424 log.cc:824] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:24.997795 18423 tablet_bootstrap.cc:492] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1: No bootstrap required, opened a new log
I20260927 12:39:24.997987 18423 ts_tablet_manager.cc:1397] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 12:39:24.999795 18424 tablet_bootstrap.cc:492] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: No bootstrap required, opened a new log
I20260927 12:39:24.999996 18424 ts_tablet_manager.cc:1397] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:39:25.001360 18423 raft_consensus.cc:348] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.001324 18422 raft_consensus.cc:348] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.001592 18422 raft_consensus.cc:374] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:25.001592 18423 raft_consensus.cc:374] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:25.001655 18422 raft_consensus.cc:729] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3564356410483ba6fc572d1706a679, State: Initialized, Role: FOLLOWER
I20260927 12:39:25.001658 18423 raft_consensus.cc:729] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e16936c78e431f897a79e8117d69b1, State: Initialized, Role: FOLLOWER
I20260927 12:39:25.002094 18423 consensus_queue.cc:260] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.002110 18422 consensus_queue.cc:260] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.002966 18274 heartbeater.cc:503] Master 127.0.98.62:44099 was elected leader, sending a full tablet report...
I20260927 12:39:25.003113 18422 ts_tablet_manager.cc:1428] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:39:25.003130 18423 ts_tablet_manager.cc:1428] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:39:25.004273 18143 heartbeater.cc:503] Master 127.0.98.62:44099 was elected leader, sending a full tablet report...
I20260927 12:39:25.004648 18424 raft_consensus.cc:348] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.004871 18424 raft_consensus.cc:374] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:25.004918 18424 raft_consensus.cc:729] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c8ccedf4db4c2cb665df4734c33e13, State: Initialized, Role: FOLLOWER
I20260927 12:39:25.005357 18424 consensus_queue.cc:260] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.006052 18405 heartbeater.cc:503] Master 127.0.98.62:44099 was elected leader, sending a full tablet report...
I20260927 12:39:25.006469 18424 ts_tablet_manager.cc:1428] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:39:25.048570 18429 raft_consensus.cc:482] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:25.048766 18429 raft_consensus.cc:504] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.049847 18429 leader_election.cc:288] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96e16936c78e431f897a79e8117d69b1 (127.0.98.1:40391), 17c8ccedf4db4c2cb665df4734c33e13 (127.0.98.3:40463)
I20260927 12:39:25.053915 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.050117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40391 (local address 127.0.98.2:56413)
0927 12:39:25.050246 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:25.050259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:25.050407 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:25.050794 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:25.050797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:25.050816 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:25.050930 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:25.050938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.051840 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.051843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:25.052777 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:25.052787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.053468 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.053471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:25.053573 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:25.053598 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:39:25.053675 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 12:39:25.053753 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:39:25.054036 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.050251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56413 (local address 127.0.98.1:40391)
0927 12:39:25.050376 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:25.050381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:25.050396 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:39:25.050545 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:25.050548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:25.050599 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:25.050673 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:25.051023 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.051712 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.052907 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.053349 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.053452 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:25.053505 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:25.053787 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 12:39:25.053882 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:39:25.054361 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.050541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53205 (local address 127.0.98.3:40463)
0927 12:39:25.050682 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:25.050685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:25.050876 (+ 191us) server_negotiation.cc:407] Connection header received
0927 12:39:25.050982 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:25.050985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:25.051036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:25.051123 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:25.051473 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.052142 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.053228 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.053737 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.053847 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:25.053905 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:25.054159 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 12:39:25.054244 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:39:25.054733 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.050406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40463 (local address 127.0.98.2:53205)
0927 12:39:25.050778 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:25.050791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:25.050891 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:25.051163 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:25.051166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:25.051183 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:25.051321 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:25.051328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.052243 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.052246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:25.053102 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:25.053112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.053889 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.053893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:25.053991 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:25.054016 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:39:25.054519 (+ 503us) client_negotiation.cc:240] Negotiation successful
0927 12:39:25.054598 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":102,"threads_started":1}
I20260927 12:39:25.054703 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2ed9797a44545a6e9ecbc0c4d6218" candidate_uuid: "1f3564356410483ba6fc572d1706a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e16936c78e431f897a79e8117d69b1" is_pre_election: true
I20260927 12:39:25.055060 18097 raft_consensus.cc:2454] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f3564356410483ba6fc572d1706a679 in term 0.
I20260927 12:39:25.055523 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2ed9797a44545a6e9ecbc0c4d6218" candidate_uuid: "1f3564356410483ba6fc572d1706a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c8ccedf4db4c2cb665df4734c33e13" is_pre_election: true
I20260927 12:39:25.055600 18161 leader_election.cc:302] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f3564356410483ba6fc572d1706a679, 96e16936c78e431f897a79e8117d69b1; no voters:
I20260927 12:39:25.055802 18359 raft_consensus.cc:2454] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f3564356410483ba6fc572d1706a679 in term 0.
I20260927 12:39:25.055907 18429 raft_consensus.cc:2781] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:39:25.056007 18429 raft_consensus.cc:482] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:39:25.056061 18429 raft_consensus.cc:3037] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:25.057127 18429 raft_consensus.cc:504] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.057480 18429 leader_election.cc:288] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [CANDIDATE]: Term 1 election: Requested vote from peers 96e16936c78e431f897a79e8117d69b1 (127.0.98.1:40391), 17c8ccedf4db4c2cb665df4734c33e13 (127.0.98.3:40463)
I20260927 12:39:25.057914 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2ed9797a44545a6e9ecbc0c4d6218" candidate_uuid: "1f3564356410483ba6fc572d1706a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c8ccedf4db4c2cb665df4734c33e13"
I20260927 12:39:25.057955 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2ed9797a44545a6e9ecbc0c4d6218" candidate_uuid: "1f3564356410483ba6fc572d1706a679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96e16936c78e431f897a79e8117d69b1"
I20260927 12:39:25.058030 18359 raft_consensus.cc:3037] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:25.058111 18097 raft_consensus.cc:3037] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:25.059109 18359 raft_consensus.cc:2454] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f3564356410483ba6fc572d1706a679 in term 1.
I20260927 12:39:25.059329 18097 raft_consensus.cc:2454] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f3564356410483ba6fc572d1706a679 in term 1.
I20260927 12:39:25.059419 18161 leader_election.cc:302] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c8ccedf4db4c2cb665df4734c33e13, 1f3564356410483ba6fc572d1706a679; no voters:
I20260927 12:39:25.059600 18429 raft_consensus.cc:2781] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:25.059871 18429 raft_consensus.cc:686] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 1 LEADER]: Becoming Leader. State: Replica: 1f3564356410483ba6fc572d1706a679, State: Running, Role: LEADER
I20260927 12:39:25.060259 18429 consensus_queue.cc:237] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } }
I20260927 12:39:25.063488 17953 catalog_manager.cc:5495] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f3564356410483ba6fc572d1706a679 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1f3564356410483ba6fc572d1706a679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f3564356410483ba6fc572d1706a679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46849 } health_report { overall_health: HEALTHY } } }
I20260927 12:39:25.080112 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.076659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.0.98.2:46849)
0927 12:39:25.076788 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:25.076792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:25.076807 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:39:25.076884 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:25.076886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:25.076941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:25.077044 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:39:25.077251 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.077741 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.078546 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:25.078751 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:25.078821 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:25.079191 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:39:25.079661 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:39:25.079760 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:25.079844 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 12:39:25.079918 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:39:25.086248 18359 raft_consensus.cc:1261] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 1 FOLLOWER]: Refusing update from remote peer 1f3564356410483ba6fc572d1706a679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:25.086431 18097 raft_consensus.cc:1261] T fff2ed9797a44545a6e9ecbc0c4d6218 P 96e16936c78e431f897a79e8117d69b1 [term 1 FOLLOWER]: Refusing update from remote peer 1f3564356410483ba6fc572d1706a679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:25.087042 18429 consensus_queue.cc:1035] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c8ccedf4db4c2cb665df4734c33e13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:25.087314 18432 consensus_queue.cc:1035] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96e16936c78e431f897a79e8117d69b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:25.096608 18436 mvcc.cc:205] Tried to move back new op lower bound from 7333940285785141248 to 7333940285689180160. Current Snapshot: MvccSnapshot[applied={T|T < 7333940285785141248}]
I20260927 12:39:25.103969 17954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:25.104251 17954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:25.107187 17954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e3b773058e74a128cc4df23a8f89752: Sending DeleteTablet for 3 replicas of tablet fff2ed9797a44545a6e9ecbc0c4d6218
I20260927 12:39:25.108404 18077 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2ed9797a44545a6e9ecbc0c4d6218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:25 UTC) from {username='slave'} at 127.0.0.1:44914
I20260927 12:39:25.108451 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2ed9797a44545a6e9ecbc0c4d6218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:25 UTC) from {username='slave'} at 127.0.0.1:47850
I20260927 12:39:25.108846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18015
I20260927 12:39:25.109215 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2ed9797a44545a6e9ecbc0c4d6218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:25 UTC) from {username='slave'} at 127.0.0.1:35472
I20260927 12:39:25.109762 18444 tablet_replica.cc:330] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: stopping tablet replica
I20260927 12:39:25.110153 18444 raft_consensus.cc:2229] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:25.110509 18444 raft_consensus.cc:2258] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:25.111428 18446 tablet_replica.cc:330] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: stopping tablet replica
I20260927 12:39:25.111799 18446 raft_consensus.cc:2229] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:39:25.112059 18444 ts_tablet_manager.cc:1905] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:25.112221 18446 raft_consensus.cc:2258] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:25.113858 18446 ts_tablet_manager.cc:1905] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:25.115079 18444 ts_tablet_manager.cc:1918] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:25.115198 18444 log.cc:1196] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/fff2ed9797a44545a6e9ecbc0c4d6218
I20260927 12:39:25.115618 18444 ts_tablet_manager.cc:1939] T fff2ed9797a44545a6e9ecbc0c4d6218 P 17c8ccedf4db4c2cb665df4734c33e13: Deleting consensus metadata
I20260927 12:39:25.117002 18446 ts_tablet_manager.cc:1918] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:25.117074 17939 catalog_manager.cc:4841] TS 17c8ccedf4db4c2cb665df4734c33e13 (127.0.98.3:40463): tablet fff2ed9797a44545a6e9ecbc0c4d6218 (table test-table [id=fc214e82bd014b4ca63a2e5446b73765]) successfully deleted
I20260927 12:39:25.117141 18446 log.cc:1196] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/fff2ed9797a44545a6e9ecbc0c4d6218
I20260927 12:39:25.117455 18446 ts_tablet_manager.cc:1939] T fff2ed9797a44545a6e9ecbc0c4d6218 P 1f3564356410483ba6fc572d1706a679: Deleting consensus metadata
W20260927 12:39:25.117475 17939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40391 (error 108)
I20260927 12:39:25.117693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18146
I20260927 12:39:25.120033 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.119717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40391 (local address 127.0.0.1:44922)
0927 12:39:25.119874 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:25.119949 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 12:39:25.120234 17939 catalog_manager.cc:4584] TS 96e16936c78e431f897a79e8117d69b1 (127.0.98.1:40391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fff2ed9797a44545a6e9ecbc0c4d6218: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40391: connect: Connection refused (error 111)
I20260927 12:39:25.123554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18277
I20260927 12:39:25.124513 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:25.123816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46849 (local address 127.0.0.1:35492)
0927 12:39:25.124310 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:25.124418 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398}
I20260927 12:39:25.128947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 17924
2026-09-27T12:39:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:25 dist-test-slave-tzgx krb5kdc[18450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:25 dist-test-slave-tzgx krb5kdc[18450](info): set up 2 sockets
Sep 27 12:39:25 dist-test-slave-tzgx krb5kdc[18450](info): commencing operation
krb5kdc: starting...
W20260927 12:39:26.119875 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:26.248215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:26.359910 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:26.360284 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:26.360347 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:26.364589 18463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:26.364662 18463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:26.364688 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:26.364711 18463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:26.364740 18463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:26.370512 18463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:26.371694 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:26.372929 18463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:26.381976 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:26.381963 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:26.381966 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:26.382266 18463 server_base.cc:1034] running on GCE node
I20260927 12:39:26.383001 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:26.384246 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:26.385440 18463 hybrid_clock.cc:648] HybridClock initialized: now 1790512766385420 us; error 35 us; skew 500 ppm
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:26.388943 18463 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:26.390182 18463 webserver.cc:460] Webserver started at http://127.0.98.62:46797/ using document root <none> and password file <none>
I20260927 12:39:26.390935 18463 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:26.391031 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:26.391264 18463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:26.393783 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "1ceac22cdb1046c5a8206ffea2c629dd"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "a831bc14f13878fb9c1c54068a3b521e"
server_key_iv: "12578c6393b97f53cf6ff0b41bcb1620"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.394361 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ceac22cdb1046c5a8206ffea2c629dd"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "a831bc14f13878fb9c1c54068a3b521e"
server_key_iv: "12578c6393b97f53cf6ff0b41bcb1620"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.398660 18463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 12:39:26.401535 18478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.403066 18463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:39:26.403218 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "1ceac22cdb1046c5a8206ffea2c629dd"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "a831bc14f13878fb9c1c54068a3b521e"
server_key_iv: "12578c6393b97f53cf6ff0b41bcb1620"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.403370 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:26.422993 18463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:26.426656 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:26.426851 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:26.434034 18463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33637
I20260927 12:39:26.434049 18530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33637 every 8 connection(s)
I20260927 12:39:26.435101 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:26.437860 18531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:26.443485 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: Bootstrap starting.
I20260927 12:39:26.445228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18463
I20260927 12:39:26.445341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:26.445596 392 external_mini_cluster.cc:1428] Setting key 821b963edb1252d1b6367e2ca0117834
I20260927 12:39:26.446050 18531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:26.446914 18531 log.cc:824] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:26.448953 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: No bootstrap required, opened a new log
I20260927 12:39:26.452527 18531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33637 } }
I20260927 12:39:26.452845 18531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:26.452924 18531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ceac22cdb1046c5a8206ffea2c629dd, State: Initialized, Role: FOLLOWER
I20260927 12:39:26.453387 18531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33637 } }
I20260927 12:39:26.453529 18531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:26.453596 18531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:26.453691 18531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:26.454892 18531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33637 } }
I20260927 12:39:26.455268 18531 leader_election.cc:302] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ceac22cdb1046c5a8206ffea2c629dd; no voters:
I20260927 12:39:26.455615 18531 leader_election.cc:288] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:26.455714 18536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:26.455940 18536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [term 1 LEADER]: Becoming Leader. State: Replica: 1ceac22cdb1046c5a8206ffea2c629dd, State: Running, Role: LEADER
I20260927 12:39:26.456338 18536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33637 } }
I20260927 12:39:26.456950 18531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:26.458055 18537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33637 } } }
I20260927 12:39:26.458190 18537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:26.458547 18538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ceac22cdb1046c5a8206ffea2c629dd. Latest consensus state: current_term: 1 leader_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ceac22cdb1046c5a8206ffea2c629dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33637 } } }
I20260927 12:39:26.458657 18538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:26.460979 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:26.446719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.0.98.62:33637)
0927 12:39:26.447288 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:26.447307 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:26.447378 (+ 71us) server_negotiation.cc:407] Connection header received
0927 12:39:26.448137 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:26.448161 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:26.448510 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:26.448889 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:26.449707 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.450764 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.451609 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.452167 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.452318 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:26.455210 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:26.455216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:26.455233 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:26.455267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:26.457464 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:26.458065 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:26.458071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:26.458077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:26.458220 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:26.458568 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:26.458571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:26.458573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:26.459131 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:26.459318 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:26.459698 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 12:39:26.460018 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":144,"threads_started":1}
W20260927 12:39:26.461971 18551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:26.462044 18551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:26.462112 18545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:26.462924 18545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:26.468341 18545 catalog_manager.cc:1347] Generated new cluster ID: 91fb0d0833e14350bfb5c62249cdb3cc
I20260927 12:39:26.468432 18545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:26.479831 18545 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:26.481244 18545 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:26.496671 18545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: Generated new TSK 0
I20260927 12:39:26.497622 18545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:39:26.527550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33637
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:26.654747 18557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:26.655035 18557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:26.655095 18557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:26.659495 18557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:26.659570 18557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:26.659636 18557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:26.665246 18557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:26.666378 18557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:26.667364 18557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:26.677198 18563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:26.677203 18562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:26.677198 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:26.677522 18557 server_base.cc:1034] running on GCE node
I20260927 12:39:26.678076 18557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:26.678819 18557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:26.680017 18557 hybrid_clock.cc:648] HybridClock initialized: now 1790512766679990 us; error 42 us; skew 500 ppm
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:26.686798 18557 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:39:26.687835 18557 webserver.cc:460] Webserver started at http://127.0.98.1:44135/ using document root <none> and password file <none>
I20260927 12:39:26.688512 18557 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:26.688596 18557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:26.688814 18557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:26.691005 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca992e41d2da48c098ff0282445c96a6"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "b0a4e26dfbc1cbbf58e66c83de11e7ec"
server_key_iv: "92a3fb49950521c541e8376071d405d2"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.691620 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca992e41d2da48c098ff0282445c96a6"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "b0a4e26dfbc1cbbf58e66c83de11e7ec"
server_key_iv: "92a3fb49950521c541e8376071d405d2"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.696362 18557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 12:39:26.699170 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.700855 18557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:26.700999 18557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "ca992e41d2da48c098ff0282445c96a6"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "b0a4e26dfbc1cbbf58e66c83de11e7ec"
server_key_iv: "92a3fb49950521c541e8376071d405d2"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.701119 18557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:26.716161 18557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:26.719228 18557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:26.719453 18557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:26.720073 18557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:26.720950 18557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:26.720999 18557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.721088 18557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:26.721119 18557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.730620 18557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39413
I20260927 12:39:26.730670 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39413 every 8 connection(s)
I20260927 12:39:26.731663 18557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:26.734952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18557
I20260927 12:39:26.735086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:39:26.735333 392 external_mini_cluster.cc:1428] Setting key 9a8ec847d1ebe19572cc46a9f43bcdc6
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:39:26.747197 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:26.733641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56629 (local address 127.0.98.62:33637)
0927 12:39:26.733805 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:26.733810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:26.734849 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 12:39:26.735874 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:26.735877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:26.735933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:26.736027 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:26.737539 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.738274 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.739954 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.740243 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.740327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:26.743401 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:26.743405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:26.743407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:26.743436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:26.745052 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:26.745631 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:26.745635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:26.745637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:26.745721 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:26.746063 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:26.746066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:26.746068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:26.746265 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:26.746360 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:26.746891 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 12:39:26.747030 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:39:26.748266 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:26.733978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33637 (local address 127.0.98.1:56629)
0927 12:39:26.734695 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:26.734739 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:26.735600 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:26.736291 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:26.736301 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:26.736847 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:26.737360 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:26.737374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.738416 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.738420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:26.739806 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:26.739818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.740391 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.740395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:26.740602 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:26.741309 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:26.741331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:26.743171 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:26.745203 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:26.745211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:26.745230 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:26.745521 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:26.745809 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:26.745812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:26.745814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:26.745974 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:26.746370 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:26.746378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:26.746655 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 12:39:26.746844 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:39:26.747197 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":121,"threads_started":1}
I20260927 12:39:26.749574 18686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33637
I20260927 12:39:26.749855 18686 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:26.750458 18686 heartbeater.cc:511] Master 127.0.98.62:33637 requested a full tablet report, sending...
I20260927 12:39:26.752199 18495 ts_manager.cc:194] Registered new tserver with Master: ca992e41d2da48c098ff0282445c96a6 (127.0.98.1:39413)
I20260927 12:39:26.753366 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:39:26.764503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33637
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:26.875384 18691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:26.875662 18691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:26.875725 18691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:26.880014 18691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:26.880116 18691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:26.880206 18691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:26.886092 18691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:26.887252 18691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:26.888353 18691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:26.898322 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:26.898324 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:26.898319 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:26.898677 18691 server_base.cc:1034] running on GCE node
I20260927 12:39:26.899220 18691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:26.899931 18691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:26.901152 18691 hybrid_clock.cc:648] HybridClock initialized: now 1790512766901123 us; error 42 us; skew 500 ppm
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:26.904557 18691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:39:26.905768 18691 webserver.cc:460] Webserver started at http://127.0.98.2:33169/ using document root <none> and password file <none>
I20260927 12:39:26.906376 18691 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:26.906452 18691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:26.906715 18691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:26.909122 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "36cca13d0f5048aab57513aa641a269b"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "9cb1956ff3ba06eb54151feca7e2838d"
server_key_iv: "0c1dec321cbfc9a4fedf29efbd2f9c0c"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.909731 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "36cca13d0f5048aab57513aa641a269b"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "9cb1956ff3ba06eb54151feca7e2838d"
server_key_iv: "0c1dec321cbfc9a4fedf29efbd2f9c0c"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.913879 18691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:26.916879 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.918350 18691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:26.918501 18691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "36cca13d0f5048aab57513aa641a269b"
format_stamp: "Formatted at 2026-09-27 12:39:26 on dist-test-slave-tzgx"
server_key: "9cb1956ff3ba06eb54151feca7e2838d"
server_key_iv: "0c1dec321cbfc9a4fedf29efbd2f9c0c"
server_key_version: "encryptionkey@0"
I20260927 12:39:26.918620 18691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:26.933284 18691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:26.936785 18691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:26.936975 18691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:26.937587 18691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:26.938458 18691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:26.938508 18691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.938573 18691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:26.938602 18691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:26.948027 18691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45627
I20260927 12:39:26.948283 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45627 every 8 connection(s)
I20260927 12:39:26.949843 18691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:26.951076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18691
I20260927 12:39:26.951191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:39:26.951490 392 external_mini_cluster.cc:1428] Setting key b69bbf45d9902cc17e3f35c68dc8a9a7
Sep 27 12:39:26 dist-test-slave-tzgx krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:39:26.967918 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:26.952229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49823 (local address 127.0.98.62:33637)
0927 12:39:26.952407 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:26.952413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:26.953918 (+ 1505us) server_negotiation.cc:407] Connection header received
0927 12:39:26.955161 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:26.955165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:26.955231 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:26.955329 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:26.956682 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.957422 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.959386 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.959687 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.959789 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:26.962764 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:26.962768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:26.962770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:26.962799 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:26.964369 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:26.965297 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:26.965304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:26.965306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:26.965506 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:26.966268 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:26.966273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:26.966274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:26.966584 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:26.966700 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:26.967509 (+ 809us) server_negotiation.cc:299] Negotiation successful
0927 12:39:26.967668 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:39:26.968662 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:26.952721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33637 (local address 127.0.98.2:49823)
0927 12:39:26.953708 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:26.953772 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:26.954842 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:26.955485 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:26.955492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:26.955914 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:26.956490 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:26.956505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.957568 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.957573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:26.959209 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:26.959227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:26.959877 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:26.959882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:26.960172 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:26.960822 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:26.960845 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:26.962555 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:26.964506 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:26.964514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:26.964534 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:26.964974 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:26.965837 (+ 863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:26.965841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:26.965843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:26.966103 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:26.966730 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:26.966738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:26.967052 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 12:39:26.967253 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 12:39:26.967576 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":165,"threads_started":1}
I20260927 12:39:26.970206 18820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33637
I20260927 12:39:26.970476 18820 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:26.971041 18820 heartbeater.cc:511] Master 127.0.98.62:33637 requested a full tablet report, sending...
I20260927 12:39:26.972325 18495 ts_manager.cc:194] Registered new tserver with Master: 36cca13d0f5048aab57513aa641a269b (127.0.98.2:45627)
I20260927 12:39:26.973132 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:39:26.983518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33637
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:27.098112 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:27.098399 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:27.098443 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:27.102630 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:27.102712 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:27.102775 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:27.108593 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:27.109829 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:27.110970 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:27.121711 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:27.121670 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:27.121670 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:27.122279 18825 server_base.cc:1034] running on GCE node
I20260927 12:39:27.122745 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:27.123595 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:27.124815 18825 hybrid_clock.cc:648] HybridClock initialized: now 1790512767124797 us; error 41 us; skew 500 ppm
Sep 27 12:39:27 dist-test-slave-tzgx krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:27.127887 18825 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:39:27.128919 18825 webserver.cc:460] Webserver started at http://127.0.98.3:37125/ using document root <none> and password file <none>
I20260927 12:39:27.129535 18825 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:27.129607 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:27.129801 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:27.131923 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "6727757cc49f44f49e28fc88c885f685"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "66b1de7d59b2bac7547881bd1ad09e57"
server_key_iv: "5d947de21bc3c37a2a973e8a3338b207"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.132577 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6727757cc49f44f49e28fc88c885f685"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "66b1de7d59b2bac7547881bd1ad09e57"
server_key_iv: "5d947de21bc3c37a2a973e8a3338b207"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.136552 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:27.139230 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:27.140715 18825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:27.140847 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "6727757cc49f44f49e28fc88c885f685"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "66b1de7d59b2bac7547881bd1ad09e57"
server_key_iv: "5d947de21bc3c37a2a973e8a3338b207"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.140964 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:27.156152 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:27.159459 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:27.159649 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:27.160276 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:27.161121 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:27.161168 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:27.161222 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:27.161248 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:27.170701 18825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36611
I20260927 12:39:27.170729 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36611 every 8 connection(s)
I20260927 12:39:27.171656 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:27.180428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 18825
I20260927 12:39:27.180584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:39:27.180862 392 external_mini_cluster.cc:1428] Setting key 4c9bf457739890ed7e52ab9730fab47d
Sep 27 12:39:27 dist-test-slave-tzgx krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:27.187776 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.173683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34079 (local address 127.0.98.62:33637)
0927 12:39:27.173871 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.173875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.174708 (+ 833us) server_negotiation.cc:407] Connection header received
0927 12:39:27.176136 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.176139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.176280 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.176380 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:27.178240 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.178826 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.180546 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.180825 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.180945 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.183932 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:27.183936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:27.183939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:27.183971 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:27.185464 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:27.186051 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:27.186054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:27.186056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:27.186138 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:27.186657 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:27.186660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:27.186661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:27.186869 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:27.186958 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.187542 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.187653 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 12:39:27.188717 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.173998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33637 (local address 127.0.98.3:34079)
0927 12:39:27.174546 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.174605 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:27.175817 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:27.176657 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:27.176668 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:27.177321 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:27.178048 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.178074 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.178998 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.179001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.180361 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.180378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.181010 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.181015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.181239 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.181959 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:27.181987 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:27.183701 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:27.185601 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:27.185608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:27.185627 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:27.185938 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:27.186270 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:27.186272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:27.186274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:27.186481 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:27.186985 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:27.186991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:27.187271 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 12:39:27.187478 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 12:39:27.187773 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260927 12:39:27.190141 18954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33637
I20260927 12:39:27.190418 18954 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:27.190918 18954 heartbeater.cc:511] Master 127.0.98.62:33637 requested a full tablet report, sending...
I20260927 12:39:27.192159 18495 ts_manager.cc:194] Registered new tserver with Master: 6727757cc49f44f49e28fc88c885f685 (127.0.98.3:36611)
I20260927 12:39:27.192747 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34079
I20260927 12:39:27.196297 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:39:27.208343 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.198488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.98.62:33637)
0927 12:39:27.198635 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.198640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.198928 (+ 288us) server_negotiation.cc:407] Connection header received
0927 12:39:27.199062 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.199066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.199112 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.199183 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:27.199923 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.200490 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.201323 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.201543 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.201634 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.203302 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:27.203306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:27.203308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:27.203337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:27.205638 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:27.206397 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:27.206401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:27.206402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:27.206488 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:27.207001 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:27.207004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:27.207005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:27.207255 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:27.207406 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.208040 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.208182 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:39:27.213357 18495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:39:27.217481 18495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:39:27.234552 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.228709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36611 (local address 127.0.0.1:48796)
0927 12:39:27.229446 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.229460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:27.229561 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:27.230095 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:27.230098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:27.230115 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:27.230201 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.230207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.231508 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.231511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.233028 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.233039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.233921 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.233925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.234075 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.234123 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 12:39:27.234234 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 12:39:27.234382 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":146,"thread_start_us":94,"threads_started":1}
I20260927 12:39:27.234556 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.228993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.98.3:36611)
0927 12:39:27.229633 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.229648 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.229683 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:39:27.229761 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.229766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.229968 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.230101 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:27.230290 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.231374 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.233161 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.233781 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.233889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.234064 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.234281 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.234404 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":115,"threads_started":1}
I20260927 12:39:27.234942 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.227989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45627 (local address 127.0.0.1:48734)
0927 12:39:27.228435 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.228496 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:27.228637 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:27.229615 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:27.229622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:27.229642 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:27.229796 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.229804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.231367 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.231371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.233027 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.233039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.234333 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.234336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.234429 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.234450 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:39:27.234500 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 12:39:27.234578 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":72,"thread_start_us":130,"threads_started":1}
I20260927 12:39:27.235688 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.228223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.0.98.2:45627)
0927 12:39:27.228583 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.228595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.228621 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:39:27.228808 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.228814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.229441 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.229946 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:27.230117 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.231245 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.233181 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.234186 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.235189 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.235342 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.235391 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.235483 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":137,"threads_started":1}
I20260927 12:39:27.235957 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.229158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.0.98.1:39413)
0927 12:39:27.229436 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.229453 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.229769 (+ 316us) server_negotiation.cc:407] Connection header received
0927 12:39:27.229959 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.229964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.230135 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.230246 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:27.230710 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.231740 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.233222 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.234180 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.234299 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.234443 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.234767 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.235808 (+ 1041us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260927 12:39:27.236363 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.228774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39413 (local address 127.0.0.1:47264)
0927 12:39:27.229689 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.229700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:27.229829 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:27.230453 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:27.230456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:27.230474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:27.230578 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.230584 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.231864 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.231867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.233070 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.233080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.234499 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.234502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.234616 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.234636 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:39:27.236134 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0927 12:39:27.236225 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260927 12:39:27.237890 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 31304a9703a94645a4eadf5272d9fbec (DEFAULT_TABLE table=test-table [id=53efc5f520874537990e7ddb3bd55f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:27.238128 18754 tablet_service.cc:1467] Processing CreateTablet for tablet 31304a9703a94645a4eadf5272d9fbec (DEFAULT_TABLE table=test-table [id=53efc5f520874537990e7ddb3bd55f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:27.239006 18888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31304a9703a94645a4eadf5272d9fbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:27.239032 18754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31304a9703a94645a4eadf5272d9fbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:27.246376 18971 tablet_bootstrap.cc:492] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Bootstrap starting.
I20260927 12:39:27.248811 18971 tablet_bootstrap.cc:654] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:27.248435 18620 tablet_service.cc:1467] Processing CreateTablet for tablet 31304a9703a94645a4eadf5272d9fbec (DEFAULT_TABLE table=test-table [id=53efc5f520874537990e7ddb3bd55f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:27.249511 18620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31304a9703a94645a4eadf5272d9fbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:27.250089 18971 log.cc:824] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:27.250700 18972 tablet_bootstrap.cc:492] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Bootstrap starting.
I20260927 12:39:27.251993 18971 tablet_bootstrap.cc:492] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: No bootstrap required, opened a new log
I20260927 12:39:27.252225 18971 ts_tablet_manager.cc:1397] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:39:27.253224 18972 tablet_bootstrap.cc:654] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:27.254559 18972 log.cc:824] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:27.255599 18971 raft_consensus.cc:348] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.255852 18971 raft_consensus.cc:374] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:27.255904 18971 raft_consensus.cc:729] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6727757cc49f44f49e28fc88c885f685, State: Initialized, Role: FOLLOWER
I20260927 12:39:27.256376 18971 consensus_queue.cc:260] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.256808 18972 tablet_bootstrap.cc:492] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: No bootstrap required, opened a new log
I20260927 12:39:27.257068 18972 ts_tablet_manager.cc:1397] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:27.257156 18954 heartbeater.cc:503] Master 127.0.98.62:33637 was elected leader, sending a full tablet report...
I20260927 12:39:27.257404 18971 ts_tablet_manager.cc:1428] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:39:27.258610 18976 tablet_bootstrap.cc:492] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: Bootstrap starting.
I20260927 12:39:27.260712 18972 raft_consensus.cc:348] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.261056 18972 raft_consensus.cc:374] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:27.261122 18972 raft_consensus.cc:729] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36cca13d0f5048aab57513aa641a269b, State: Initialized, Role: FOLLOWER
I20260927 12:39:27.261219 18976 tablet_bootstrap.cc:654] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:27.261617 18972 consensus_queue.cc:260] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.262113 18976 log.cc:824] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:27.262678 18820 heartbeater.cc:503] Master 127.0.98.62:33637 was elected leader, sending a full tablet report...
I20260927 12:39:27.263895 18972 ts_tablet_manager.cc:1428] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:27.264472 18976 tablet_bootstrap.cc:492] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: No bootstrap required, opened a new log
I20260927 12:39:27.264653 18976 ts_tablet_manager.cc:1397] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:39:27.268623 18976 raft_consensus.cc:348] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.268958 18976 raft_consensus.cc:374] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:27.269014 18976 raft_consensus.cc:729] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca992e41d2da48c098ff0282445c96a6, State: Initialized, Role: FOLLOWER
I20260927 12:39:27.269527 18976 consensus_queue.cc:260] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.270335 18686 heartbeater.cc:503] Master 127.0.98.62:33637 was elected leader, sending a full tablet report...
I20260927 12:39:27.270529 18976 ts_tablet_manager.cc:1428] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 12:39:27.422688 18955 tablet.cc:2369] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:27.432999 18979 raft_consensus.cc:482] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:27.433192 18979 raft_consensus.cc:504] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.434432 18979 leader_election.cc:288] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36cca13d0f5048aab57513aa641a269b (127.0.98.2:45627), 6727757cc49f44f49e28fc88c885f685 (127.0.98.3:36611)
I20260927 12:39:27.439422 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.435157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49547 (local address 127.0.98.3:36611)
0927 12:39:27.435286 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.435290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.435449 (+ 159us) server_negotiation.cc:407] Connection header received
0927 12:39:27.435565 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.435568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.435613 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.435656 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:27.435992 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.436536 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.438273 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.438870 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.438976 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.439044 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.439223 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.439302 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:39:27.439472 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.434692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36611 (local address 127.0.98.1:49547)
0927 12:39:27.435359 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.435372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:27.435456 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:27.435775 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:27.435778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:27.435794 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:27.435886 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.435890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.436637 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.436641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.438132 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.438152 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.438999 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.439002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.439081 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.439104 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:39:27.439192 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 12:39:27.439268 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":110,"thread_start_us":140,"threads_started":1}
I20260927 12:39:27.440323 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.434668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45627 (local address 127.0.98.1:50339)
0927 12:39:27.435117 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.435132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:27.435261 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:27.435549 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:27.435551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:27.435565 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:27.435707 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.435714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.436709 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.436712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.438132 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:27.438145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.439880 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.439884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:27.439968 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.439987 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:39:27.440114 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 12:39:27.440187 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":64,"thread_start_us":167,"threads_started":1}
I20260927 12:39:27.440340 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.434800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50339 (local address 127.0.98.2:45627)
0927 12:39:27.434942 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.434946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.435320 (+ 374us) server_negotiation.cc:407] Connection header received
0927 12:39:27.435382 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.435384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.435434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.435541 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:27.435804 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.436612 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.439107 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.439741 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.439838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.439896 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.440133 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.440217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 12:39:27.440768 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31304a9703a94645a4eadf5272d9fbec" candidate_uuid: "ca992e41d2da48c098ff0282445c96a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6727757cc49f44f49e28fc88c885f685" is_pre_election: true
I20260927 12:39:27.441088 18908 raft_consensus.cc:2454] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992e41d2da48c098ff0282445c96a6 in term 0.
I20260927 12:39:27.441593 18574 leader_election.cc:302] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6727757cc49f44f49e28fc88c885f685, ca992e41d2da48c098ff0282445c96a6; no voters:
I20260927 12:39:27.441824 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31304a9703a94645a4eadf5272d9fbec" candidate_uuid: "ca992e41d2da48c098ff0282445c96a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36cca13d0f5048aab57513aa641a269b" is_pre_election: true
I20260927 12:39:27.441901 18979 raft_consensus.cc:2781] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:39:27.441984 18979 raft_consensus.cc:482] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:39:27.442030 18979 raft_consensus.cc:3037] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:27.442132 18774 raft_consensus.cc:2454] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca992e41d2da48c098ff0282445c96a6 in term 0.
I20260927 12:39:27.443295 18979 raft_consensus.cc:504] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.443691 18979 leader_election.cc:288] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [CANDIDATE]: Term 1 election: Requested vote from peers 36cca13d0f5048aab57513aa641a269b (127.0.98.2:45627), 6727757cc49f44f49e28fc88c885f685 (127.0.98.3:36611)
I20260927 12:39:27.444056 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31304a9703a94645a4eadf5272d9fbec" candidate_uuid: "ca992e41d2da48c098ff0282445c96a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36cca13d0f5048aab57513aa641a269b"
I20260927 12:39:27.444067 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31304a9703a94645a4eadf5272d9fbec" candidate_uuid: "ca992e41d2da48c098ff0282445c96a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6727757cc49f44f49e28fc88c885f685"
I20260927 12:39:27.444207 18908 raft_consensus.cc:3037] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:27.444212 18774 raft_consensus.cc:3037] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:27.445466 18774 raft_consensus.cc:2454] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992e41d2da48c098ff0282445c96a6 in term 1.
I20260927 12:39:27.445523 18908 raft_consensus.cc:2454] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca992e41d2da48c098ff0282445c96a6 in term 1.
I20260927 12:39:27.445833 18574 leader_election.cc:302] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36cca13d0f5048aab57513aa641a269b, 6727757cc49f44f49e28fc88c885f685, ca992e41d2da48c098ff0282445c96a6; no voters:
I20260927 12:39:27.446022 18979 raft_consensus.cc:2781] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:27.446235 18979 raft_consensus.cc:686] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [term 1 LEADER]: Becoming Leader. State: Replica: ca992e41d2da48c098ff0282445c96a6, State: Running, Role: LEADER
I20260927 12:39:27.446609 18979 consensus_queue.cc:237] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } }
I20260927 12:39:27.449745 18495 catalog_manager.cc:5495] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca992e41d2da48c098ff0282445c96a6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ca992e41d2da48c098ff0282445c96a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca992e41d2da48c098ff0282445c96a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 } health_report { overall_health: UNKNOWN } } }
W20260927 12:39:27.451364 18821 tablet.cc:2369] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:39:27.483136 18687 tablet.cc:2369] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:27.483984 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.480290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.0.98.1:39413)
0927 12:39:27.480433 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.480437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.480452 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:39:27.480628 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.480631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.480692 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.480789 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:39:27.480997 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.481509 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.482410 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.482643 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.482736 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.483058 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:39:27.483549 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:39:27.483662 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.483782 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.483821 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:39:27.489960 18774 raft_consensus.cc:1261] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 1 FOLLOWER]: Refusing update from remote peer ca992e41d2da48c098ff0282445c96a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:27.489960 18908 raft_consensus.cc:1261] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 1 FOLLOWER]: Refusing update from remote peer ca992e41d2da48c098ff0282445c96a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:27.490694 18982 consensus_queue.cc:1035] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36cca13d0f5048aab57513aa641a269b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:27.490911 18979 consensus_queue.cc:1035] T 31304a9703a94645a4eadf5272d9fbec P ca992e41d2da48c098ff0282445c96a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6727757cc49f44f49e28fc88c885f685" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:27.498170 18985 mvcc.cc:205] Tried to move back new op lower bound from 7333940295630897152 to 7333940295463940096. Current Snapshot: MvccSnapshot[applied={T|T < 7333940295630897152}]
I20260927 12:39:27.500921 18986 mvcc.cc:205] Tried to move back new op lower bound from 7333940295630897152 to 7333940295463940096. Current Snapshot: MvccSnapshot[applied={T|T < 7333940295630897152}]
I20260927 12:39:27.508407 18495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:27.508643 18495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:27.511448 18495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ceac22cdb1046c5a8206ffea2c629dd: Sending DeleteTablet for 3 replicas of tablet 31304a9703a94645a4eadf5272d9fbec
I20260927 12:39:27.512403 18754 tablet_service.cc:1514] Processing DeleteTablet for tablet 31304a9703a94645a4eadf5272d9fbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48734
I20260927 12:39:27.512617 18620 tablet_service.cc:1514] Processing DeleteTablet for tablet 31304a9703a94645a4eadf5272d9fbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47264
I20260927 12:39:27.512974 18994 tablet_replica.cc:330] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: stopping tablet replica
I20260927 12:39:27.513227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18557
I20260927 12:39:27.513319 18994 raft_consensus.cc:2229] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:27.513388 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet 31304a9703a94645a4eadf5272d9fbec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48796
I20260927 12:39:27.513651 18994 raft_consensus.cc:2258] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:27.514599 18996 tablet_replica.cc:330] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: stopping tablet replica
I20260927 12:39:27.514824 18996 raft_consensus.cc:2229] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:27.515065 18996 raft_consensus.cc:2258] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:27.515120 18994 ts_tablet_manager.cc:1905] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:27.516314 18996 ts_tablet_manager.cc:1905] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:27.517822 18994 ts_tablet_manager.cc:1918] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:27.517915 18994 log.cc:1196] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/31304a9703a94645a4eadf5272d9fbec
I20260927 12:39:27.518175 18994 ts_tablet_manager.cc:1939] T 31304a9703a94645a4eadf5272d9fbec P 36cca13d0f5048aab57513aa641a269b: Deleting consensus metadata
I20260927 12:39:27.518898 18996 ts_tablet_manager.cc:1918] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:27.519002 18996 log.cc:1196] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/31304a9703a94645a4eadf5272d9fbec
I20260927 12:39:27.519284 18996 ts_tablet_manager.cc:1939] T 31304a9703a94645a4eadf5272d9fbec P 6727757cc49f44f49e28fc88c885f685: Deleting consensus metadata
I20260927 12:39:27.519454 18479 catalog_manager.cc:4841] TS 36cca13d0f5048aab57513aa641a269b (127.0.98.2:45627): tablet 31304a9703a94645a4eadf5272d9fbec (table test-table [id=53efc5f520874537990e7ddb3bd55f06]) successfully deleted
I20260927 12:39:27.520457 18480 catalog_manager.cc:4841] TS 6727757cc49f44f49e28fc88c885f685 (127.0.98.3:36611): tablet 31304a9703a94645a4eadf5272d9fbec (table test-table [id=53efc5f520874537990e7ddb3bd55f06]) successfully deleted
I20260927 12:39:27.521908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18691
W20260927 12:39:27.521935 18479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39413 (error 108)
I20260927 12:39:27.525096 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.524775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39413 (local address 127.0.0.1:47276)
0927 12:39:27.524942 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:27.525012 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 12:39:27.525282 18479 catalog_manager.cc:4584] TS ca992e41d2da48c098ff0282445c96a6 (127.0.98.1:39413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31304a9703a94645a4eadf5272d9fbec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39413: connect: Connection refused (error 111)
I20260927 12:39:27.527340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18825
I20260927 12:39:27.532596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 18463
2026-09-27T12:39:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2404 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T12:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:27Z Disabled control of system clock
I20260927 12:39:27.562124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38499
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:39:27.677863 19001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:27.678140 19001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:27.678176 19001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:27.682210 19001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:27.682276 19001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:27.682288 19001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:27.682304 19001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:27.682318 19001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:27.687211 19001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:27.688351 19001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:27.689749 19001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:27.698387 19009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:27.698374 19007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:27.698374 19006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:27.698729 19001 server_base.cc:1034] running on GCE node
I20260927 12:39:27.699503 19001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:27.700735 19001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:27.701920 19001 hybrid_clock.cc:648] HybridClock initialized: now 1790512767701897 us; error 38 us; skew 500 ppm
I20260927 12:39:27.703886 19001 webserver.cc:460] Webserver started at http://127.0.98.62:40739/ using document root <none> and password file <none>
I20260927 12:39:27.704496 19001 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:27.704571 19001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:27.704768 19001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:27.706849 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "f5a01bb92b6a4b3286644844d77a90e6"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "9d9f7e21749a23ee71a6cc811f429efc"
server_key_iv: "68ffa52ebb8d30b2fb36a496a48527e3"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.707737 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5a01bb92b6a4b3286644844d77a90e6"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "9d9f7e21749a23ee71a6cc811f429efc"
server_key_iv: "68ffa52ebb8d30b2fb36a496a48527e3"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.712699 19001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:27.715555 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:27.717195 19001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:27.717351 19001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "f5a01bb92b6a4b3286644844d77a90e6"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "9d9f7e21749a23ee71a6cc811f429efc"
server_key_iv: "68ffa52ebb8d30b2fb36a496a48527e3"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.717474 19001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:27.747455 19001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:27.748303 19001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:27.748489 19001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:27.756330 19001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38499
I20260927 12:39:27.756387 19067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38499 every 8 connection(s)
I20260927 12:39:27.757337 19001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:27.758571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19001
I20260927 12:39:27.758762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:27.759071 392 external_mini_cluster.cc:1428] Setting key b7b5540b5eb009c45b8ce6ab3568b4d6
I20260927 12:39:27.761085 19068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:27.767238 19068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6: Bootstrap starting.
I20260927 12:39:27.768908 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:27.760629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.62:38499)
0927 12:39:27.761356 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:27.761373 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:27.761417 (+ 44us) server_negotiation.cc:407] Connection header received
0927 12:39:27.762119 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:27.762136 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:27.762564 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:27.762908 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:27.763364 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.764431 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.765364 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:27.765867 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:27.766032 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:27.767050 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:27.767055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:27.767070 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:27.767509 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:27.767572 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:27.767596 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:27.767688 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:27.767827 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 12:39:27.768149 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260927 12:39:27.769810 19068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:27.770599 19068 log.cc:824] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:27.772771 19068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6: No bootstrap required, opened a new log
I20260927 12:39:27.776301 19068 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a01bb92b6a4b3286644844d77a90e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38499 } }
I20260927 12:39:27.776551 19068 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:27.776587 19068 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a01bb92b6a4b3286644844d77a90e6, State: Initialized, Role: FOLLOWER
I20260927 12:39:27.777045 19068 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a01bb92b6a4b3286644844d77a90e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38499 } }
I20260927 12:39:27.777202 19068 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:27.777268 19068 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:27.777360 19068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:27.778317 19068 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a01bb92b6a4b3286644844d77a90e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38499 } }
I20260927 12:39:27.778638 19068 leader_election.cc:302] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5a01bb92b6a4b3286644844d77a90e6; no voters:
I20260927 12:39:27.778903 19068 leader_election.cc:288] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:27.779018 19073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:27.779213 19073 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [term 1 LEADER]: Becoming Leader. State: Replica: f5a01bb92b6a4b3286644844d77a90e6, State: Running, Role: LEADER
I20260927 12:39:27.779552 19073 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a01bb92b6a4b3286644844d77a90e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38499 } }
I20260927 12:39:27.780071 19068 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:27.781179 19075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5a01bb92b6a4b3286644844d77a90e6. Latest consensus state: current_term: 1 leader_uuid: "f5a01bb92b6a4b3286644844d77a90e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a01bb92b6a4b3286644844d77a90e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38499 } } }
I20260927 12:39:27.781311 19075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:27.781549 19074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5a01bb92b6a4b3286644844d77a90e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5a01bb92b6a4b3286644844d77a90e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38499 } } }
I20260927 12:39:27.781702 19074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:27.781776 19082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:27.784907 19082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:27.791128 19082 catalog_manager.cc:1347] Generated new cluster ID: 1e9b263903494156a2380013b5966091
I20260927 12:39:27.791210 19082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:27.809279 19082 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:27.810736 19082 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:27.824285 19082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6: Generated new TSK 0
I20260927 12:39:27.825232 19082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:27.829648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38499
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:39:27.948309 19092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:27.948596 19092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:27.948637 19092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:27.952880 19092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:27.952951 19092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:27.953022 19092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:27.958083 19092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:27.959192 19092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:27.960574 19092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:27.970028 19100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:27.970047 19098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:27.970028 19097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:27.970250 19092 server_base.cc:1034] running on GCE node
I20260927 12:39:27.970793 19092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:27.971451 19092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:27.972602 19092 hybrid_clock.cc:648] HybridClock initialized: now 1790512767972589 us; error 30 us; skew 500 ppm
I20260927 12:39:27.974684 19092 webserver.cc:460] Webserver started at http://127.0.98.1:32787/ using document root <none> and password file <none>
I20260927 12:39:27.975260 19092 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:27.975322 19092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:27.975493 19092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:27.977672 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "f53064e78fd54ee88c6a33450ca30b5d"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "fd55c1588acb81dd4c2c98357c9f6890"
server_key_iv: "a8fdaf7b6a0dd28d76a41d66eb9d4c2c"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.978279 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f53064e78fd54ee88c6a33450ca30b5d"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "fd55c1588acb81dd4c2c98357c9f6890"
server_key_iv: "a8fdaf7b6a0dd28d76a41d66eb9d4c2c"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.982285 19092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:27.984961 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:27.986258 19092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:27.986379 19092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "f53064e78fd54ee88c6a33450ca30b5d"
format_stamp: "Formatted at 2026-09-27 12:39:27 on dist-test-slave-tzgx"
server_key: "fd55c1588acb81dd4c2c98357c9f6890"
server_key_iv: "a8fdaf7b6a0dd28d76a41d66eb9d4c2c"
server_key_version: "encryptionkey@0"
I20260927 12:39:27.986495 19092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:28.001098 19092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:28.001816 19092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:28.001995 19092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:28.002635 19092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:28.003621 19092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:28.003670 19092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.003727 19092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:28.003755 19092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.013412 19092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35399
I20260927 12:39:28.013459 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35399 every 8 connection(s)
I20260927 12:39:28.014394 19092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:28.016763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19092
I20260927 12:39:28.016978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:39:28.017292 392 external_mini_cluster.cc:1428] Setting key d77feb72a0e1abf76606b21f56b542ba
I20260927 12:39:28.019462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38499
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:39:28.025302 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.016706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36095 (local address 127.0.98.62:38499)
0927 12:39:28.016946 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.016952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.018081 (+ 1129us) server_negotiation.cc:407] Connection header received
0927 12:39:28.019720 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.019724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.019789 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.019833 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:28.021139 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.021686 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.023005 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.023248 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.023346 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.024748 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:28.024753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:28.024755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:28.024778 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:28.024803 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:28.024813 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:28.024876 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.025042 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.025143 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 12:39:28.026249 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.017149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38499 (local address 127.0.98.1:36095)
0927 12:39:28.017928 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.017984 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.019431 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.020250 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.020259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.020357 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:28.020954 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.020971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.021840 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.021844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.022862 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.022876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.023428 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.023432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.023710 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.024430 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:28.024431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:28.024481 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:28.024564 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:28.024899 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:28.024910 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:28.024916 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.025034 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.025369 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":158,"threads_started":1}
I20260927 12:39:28.027968 19220 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38499
I20260927 12:39:28.028249 19220 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:28.028843 19220 heartbeater.cc:511] Master 127.0.98.62:38499 requested a full tablet report, sending...
I20260927 12:39:28.030514 19032 ts_manager.cc:194] Registered new tserver with Master: f53064e78fd54ee88c6a33450ca30b5d (127.0.98.1:35399)
I20260927 12:39:28.031708 19032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36095
W20260927 12:39:28.137068 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:28.137370 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:28.137429 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:28.141723 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:28.141798 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:28.141876 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:28.147028 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:28.148177 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:28.149641 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:28.159953 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:28.160070 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:28.159982 19223 server_base.cc:1034] running on GCE node
W20260927 12:39:28.159967 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:28.160727 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:28.161450 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:28.162732 19223 hybrid_clock.cc:648] HybridClock initialized: now 1790512768162674 us; error 72 us; skew 500 ppm
I20260927 12:39:28.164865 19223 webserver.cc:460] Webserver started at http://127.0.98.2:45955/ using document root <none> and password file <none>
I20260927 12:39:28.165506 19223 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:28.165585 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:28.165792 19223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:28.167974 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "0dad4f3a43e342aa82dc9ba8143713a1"
format_stamp: "Formatted at 2026-09-27 12:39:28 on dist-test-slave-tzgx"
server_key: "5b529433b23d4111b99a2d4516519b6f"
server_key_iv: "6c7e0a767b19d2865e21a7376317f354"
server_key_version: "encryptionkey@0"
I20260927 12:39:28.168598 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0dad4f3a43e342aa82dc9ba8143713a1"
format_stamp: "Formatted at 2026-09-27 12:39:28 on dist-test-slave-tzgx"
server_key: "5b529433b23d4111b99a2d4516519b6f"
server_key_iv: "6c7e0a767b19d2865e21a7376317f354"
server_key_version: "encryptionkey@0"
I20260927 12:39:28.173205 19223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 12:39:28.176285 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.177768 19223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:39:28.177911 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "0dad4f3a43e342aa82dc9ba8143713a1"
format_stamp: "Formatted at 2026-09-27 12:39:28 on dist-test-slave-tzgx"
server_key: "5b529433b23d4111b99a2d4516519b6f"
server_key_iv: "6c7e0a767b19d2865e21a7376317f354"
server_key_version: "encryptionkey@0"
I20260927 12:39:28.178050 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:28.191186 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:28.191941 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:28.192170 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:28.192771 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:28.193715 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:28.193763 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.193804 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:28.193816 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.201984 19223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34863
I20260927 12:39:28.202013 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34863 every 8 connection(s)
I20260927 12:39:28.202885 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:28.206223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19223
I20260927 12:39:28.206362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:39:28.206657 392 external_mini_cluster.cc:1428] Setting key 7178be1998176b3b93b0076f3c7bb145
I20260927 12:39:28.208896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38499
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:39:28.213331 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.204769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39151 (local address 127.0.98.62:38499)
0927 12:39:28.204937 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.204942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.206093 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 12:39:28.207731 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.207734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.207782 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.207878 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:28.209192 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.209728 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.211104 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.211413 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.211527 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.212774 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:28.212779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:28.212781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:28.212799 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:28.212838 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:28.212849 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:28.212934 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.213086 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.213168 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:39:28.214167 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.205171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38499 (local address 127.0.98.2:39151)
0927 12:39:28.205898 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.205946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.207518 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.208125 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.208135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.208250 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:28.208984 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.209006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.209886 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.209889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.210973 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.210985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.211602 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.211606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.211766 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.212471 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:28.212473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:28.212518 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:28.212592 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:28.212957 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:28.212963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:28.212967 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.213063 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.213310 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":115,"threads_started":1}
I20260927 12:39:28.215314 19351 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38499
I20260927 12:39:28.215581 19351 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:28.216298 19351 heartbeater.cc:511] Master 127.0.98.62:38499 requested a full tablet report, sending...
I20260927 12:39:28.217567 19032 ts_manager.cc:194] Registered new tserver with Master: 0dad4f3a43e342aa82dc9ba8143713a1 (127.0.98.2:34863)
I20260927 12:39:28.218173 19032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39151
W20260927 12:39:28.322288 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:28.322604 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:28.322649 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:28.326973 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:28.327045 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:28.327105 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:28.332062 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:28.333240 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:28.334735 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:28.344502 19362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:28.344592 19359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:28.344514 19360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:28.344746 19354 server_base.cc:1034] running on GCE node
I20260927 12:39:28.345268 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:28.345914 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:28.347092 19354 hybrid_clock.cc:648] HybridClock initialized: now 1790512768347074 us; error 33 us; skew 500 ppm
I20260927 12:39:28.349121 19354 webserver.cc:460] Webserver started at http://127.0.98.3:38447/ using document root <none> and password file <none>
I20260927 12:39:28.349697 19354 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:28.349764 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:28.349964 19354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:28.352072 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "174ef8933ded47aea5cce72788984cb2"
format_stamp: "Formatted at 2026-09-27 12:39:28 on dist-test-slave-tzgx"
server_key: "e7eae5edc4a01fb18f4ec0d0b502d597"
server_key_iv: "8400e4d1f1f5cccf5cb0627b0e7b3653"
server_key_version: "encryptionkey@0"
I20260927 12:39:28.352675 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "174ef8933ded47aea5cce72788984cb2"
format_stamp: "Formatted at 2026-09-27 12:39:28 on dist-test-slave-tzgx"
server_key: "e7eae5edc4a01fb18f4ec0d0b502d597"
server_key_iv: "8400e4d1f1f5cccf5cb0627b0e7b3653"
server_key_version: "encryptionkey@0"
I20260927 12:39:28.357488 19354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 12:39:28.360646 19368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.362306 19354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:39:28.362433 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "174ef8933ded47aea5cce72788984cb2"
format_stamp: "Formatted at 2026-09-27 12:39:28 on dist-test-slave-tzgx"
server_key: "e7eae5edc4a01fb18f4ec0d0b502d597"
server_key_iv: "8400e4d1f1f5cccf5cb0627b0e7b3653"
server_key_version: "encryptionkey@0"
I20260927 12:39:28.362567 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:28.379192 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:28.379889 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:28.380050 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:28.380693 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:28.381681 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:28.381727 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.381764 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:28.381785 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:28.390161 19354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43115
I20260927 12:39:28.390213 19481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43115 every 8 connection(s)
I20260927 12:39:28.391165 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:28.395592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19354
I20260927 12:39:28.395712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:39:28.395969 392 external_mini_cluster.cc:1428] Setting key cdc0cfc7ee8a359ba564eafa9f28ffbd
I20260927 12:39:28.402184 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.393463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39047 (local address 127.0.98.62:38499)
0927 12:39:28.393653 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.393658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.395000 (+ 1342us) server_negotiation.cc:407] Connection header received
0927 12:39:28.396599 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.396603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.396662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.396885 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:28.398097 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.398728 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.400119 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.400354 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.400442 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.401629 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:28.401633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:28.401635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:28.401651 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:28.401675 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:28.401683 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:28.401751 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.401978 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.402055 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 12:39:28.403158 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.394040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38499 (local address 127.0.98.3:39047)
0927 12:39:28.394818 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.394865 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.396318 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.397099 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.397109 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.397226 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:28.397915 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.397939 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.398874 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.398877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.399934 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.399948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.400499 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.400502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.400678 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.401299 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:28.401301 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:28.401348 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:28.401428 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:28.401792 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:28.401800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:28.401805 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.401937 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.402225 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":140,"threads_started":1}
I20260927 12:39:28.404536 19482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38499
I20260927 12:39:28.404790 19482 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:28.405349 19482 heartbeater.cc:511] Master 127.0.98.62:38499 requested a full tablet report, sending...
I20260927 12:39:28.406507 19032 ts_manager.cc:194] Registered new tserver with Master: 174ef8933ded47aea5cce72788984cb2 (127.0.98.3:43115)
I20260927 12:39:28.407088 19032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39047
I20260927 12:39:28.410615 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:39:28.417508 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.413237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.98.62:38499)
0927 12:39:28.413384 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.413387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.413502 (+ 115us) server_negotiation.cc:407] Connection header received
0927 12:39:28.413607 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.413609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.413655 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.413722 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:28.414087 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.414671 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.415576 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.415789 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.415869 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.416948 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:28.416951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:28.416953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:28.416968 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:28.416990 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:28.416999 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:28.417123 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.417344 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.417385 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 12:39:28.420979 19032 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:39:28.424028 19032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:39:28.439184 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.433048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.0.1:56014)
0927 12:39:28.433493 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.433549 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.433677 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.434724 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.434730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.434765 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:28.434928 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.434936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.436130 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.436133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.437472 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.437483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.438559 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.438563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.438683 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.438733 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.438853 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.439012 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":7,"thread_start_us":81,"threads_started":1}
I20260927 12:39:28.439390 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.433406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34863 (local address 127.0.0.1:42256)
0927 12:39:28.433811 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.433825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.433907 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.434950 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.434954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.434972 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:28.435091 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.435098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.436488 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.436492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.437680 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.437688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.438744 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.438748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.438836 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.438856 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.438897 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.439274 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":395,"thread_start_us":116,"threads_started":1}
I20260927 12:39:28.439491 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.433803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.0.98.2:34863)
0927 12:39:28.434211 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.434553 (+ 342us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.434580 (+ 27us) server_negotiation.cc:407] Connection header received
0927 12:39:28.434657 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.434661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.434813 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.434951 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:28.435200 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.436362 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.437797 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.438603 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.438981 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.439174 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.439227 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.439324 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":108,"threads_started":1}
I20260927 12:39:28.439790 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.434159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.0.98.3:43115)
0927 12:39:28.434574 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.434585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.434608 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:39:28.434676 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.434681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.434853 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.435019 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:28.435394 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.436514 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.437964 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.438711 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.438808 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.438974 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.439067 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.439168 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":145,"threads_started":1}
I20260927 12:39:28.439831 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.433220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.0.98.1:35399)
0927 12:39:28.434165 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.434177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.434202 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:39:28.434279 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.434285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.434547 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.435078 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:28.435214 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.435994 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.437615 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.438406 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.438519 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.438741 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.439570 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.439681 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":126,"threads_started":1}
I20260927 12:39:28.440037 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.433756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43115 (local address 127.0.0.1:37392)
0927 12:39:28.434355 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.434367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.434444 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.435180 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.435183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.435201 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:28.435273 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.435278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.436612 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.436615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.437837 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.437847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.438820 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.438823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.438908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.438956 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.439036 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.439113 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260927 12:39:28.442682 19416 tablet_service.cc:1467] Processing CreateTablet for tablet 83315d30e8ab49bd8255fbb4491467a3 (DEFAULT_TABLE table=test-table [id=f693b94ce5764f0faacd5b76735e8a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:28.442690 19285 tablet_service.cc:1467] Processing CreateTablet for tablet 83315d30e8ab49bd8255fbb4491467a3 (DEFAULT_TABLE table=test-table [id=f693b94ce5764f0faacd5b76735e8a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:28.442690 19154 tablet_service.cc:1467] Processing CreateTablet for tablet 83315d30e8ab49bd8255fbb4491467a3 (DEFAULT_TABLE table=test-table [id=f693b94ce5764f0faacd5b76735e8a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:28.443707 19416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83315d30e8ab49bd8255fbb4491467a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:28.443722 19154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83315d30e8ab49bd8255fbb4491467a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:28.443722 19285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83315d30e8ab49bd8255fbb4491467a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:28.450721 19499 tablet_bootstrap.cc:492] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Bootstrap starting.
I20260927 12:39:28.451618 19500 tablet_bootstrap.cc:492] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: Bootstrap starting.
I20260927 12:39:28.452452 19501 tablet_bootstrap.cc:492] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: Bootstrap starting.
I20260927 12:39:28.453146 19499 tablet_bootstrap.cc:654] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:28.453514 19500 tablet_bootstrap.cc:654] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:28.454049 19499 log.cc:824] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:28.454195 19500 log.cc:824] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:28.454531 19501 tablet_bootstrap.cc:654] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:28.455361 19501 log.cc:824] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:28.456432 19499 tablet_bootstrap.cc:492] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: No bootstrap required, opened a new log
I20260927 12:39:28.456676 19499 ts_tablet_manager.cc:1397] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:39:28.456995 19500 tablet_bootstrap.cc:492] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: No bootstrap required, opened a new log
I20260927 12:39:28.457216 19500 ts_tablet_manager.cc:1397] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:39:28.457331 19501 tablet_bootstrap.cc:492] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: No bootstrap required, opened a new log
I20260927 12:39:28.457523 19501 ts_tablet_manager.cc:1397] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 12:39:28.460744 19501 raft_consensus.cc:348] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.461014 19501 raft_consensus.cc:374] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:28.461066 19501 raft_consensus.cc:729] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dad4f3a43e342aa82dc9ba8143713a1, State: Initialized, Role: FOLLOWER
I20260927 12:39:28.461073 19500 raft_consensus.cc:348] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.461074 19499 raft_consensus.cc:348] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.461339 19500 raft_consensus.cc:374] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:28.461350 19499 raft_consensus.cc:374] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:28.461395 19500 raft_consensus.cc:729] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53064e78fd54ee88c6a33450ca30b5d, State: Initialized, Role: FOLLOWER
I20260927 12:39:28.461395 19499 raft_consensus.cc:729] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174ef8933ded47aea5cce72788984cb2, State: Initialized, Role: FOLLOWER
I20260927 12:39:28.461563 19501 consensus_queue.cc:260] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.461824 19500 consensus_queue.cc:260] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.461824 19499 consensus_queue.cc:260] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.462333 19351 heartbeater.cc:503] Master 127.0.98.62:38499 was elected leader, sending a full tablet report...
I20260927 12:39:28.462852 19499 ts_tablet_manager.cc:1428] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:39:28.462857 19500 ts_tablet_manager.cc:1428] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:39:28.463292 19482 heartbeater.cc:503] Master 127.0.98.62:38499 was elected leader, sending a full tablet report...
I20260927 12:39:28.463351 19220 heartbeater.cc:503] Master 127.0.98.62:38499 was elected leader, sending a full tablet report...
I20260927 12:39:28.465026 19501 ts_tablet_manager.cc:1428] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
W20260927 12:39:28.515671 19221 tablet.cc:2369] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:39:28.642493 19483 tablet.cc:2369] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:28.668702 19507 raft_consensus.cc:482] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:28.668903 19507 raft_consensus.cc:504] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.670770 19507 leader_election.cc:288] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f53064e78fd54ee88c6a33450ca30b5d (127.0.98.1:35399), 0dad4f3a43e342aa82dc9ba8143713a1 (127.0.98.2:34863)
I20260927 12:39:28.674090 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.670626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54953 (local address 127.0.98.2:34863)
0927 12:39:28.670731 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.670736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.671003 (+ 267us) server_negotiation.cc:407] Connection header received
0927 12:39:28.671110 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.671113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.671164 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.671238 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:28.671567 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.672144 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.673167 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.673579 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.673682 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.673743 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.673892 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.673959 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:39:28.674114 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.670454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.98.3:60309)
0927 12:39:28.670630 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.670647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.670734 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.671117 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.671120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.671137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:28.671252 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.671260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.672042 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.672046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.673004 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.673014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.673659 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.673663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.673760 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.673784 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.673850 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.673946 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":38}
I20260927 12:39:28.674108 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.670558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60309 (local address 127.0.98.1:35399)
0927 12:39:28.670685 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.670690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.670705 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:39:28.670875 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.670879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.670937 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.671003 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:28.671378 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.671915 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.673141 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.673541 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.673625 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.673692 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.673887 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.673956 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:39:28.674134 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.670460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34863 (local address 127.0.98.3:54953)
0927 12:39:28.670904 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.670918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:28.671028 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:28.671243 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:28.671244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:28.671255 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:28.671371 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.671376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.672251 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.672254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.673035 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:28.673045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.673697 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.673701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:28.673783 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.673801 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:39:28.673869 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:39:28.673951 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":112,"thread_start_us":121,"threads_started":1}
I20260927 12:39:28.674798 19174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83315d30e8ab49bd8255fbb4491467a3" candidate_uuid: "174ef8933ded47aea5cce72788984cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53064e78fd54ee88c6a33450ca30b5d" is_pre_election: true
I20260927 12:39:28.674803 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83315d30e8ab49bd8255fbb4491467a3" candidate_uuid: "174ef8933ded47aea5cce72788984cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" is_pre_election: true
I20260927 12:39:28.675105 19305 raft_consensus.cc:2454] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174ef8933ded47aea5cce72788984cb2 in term 0.
I20260927 12:39:28.675110 19174 raft_consensus.cc:2454] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 174ef8933ded47aea5cce72788984cb2 in term 0.
I20260927 12:39:28.675594 19371 leader_election.cc:302] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dad4f3a43e342aa82dc9ba8143713a1, 174ef8933ded47aea5cce72788984cb2; no voters:
I20260927 12:39:28.675892 19507 raft_consensus.cc:2781] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:39:28.675976 19507 raft_consensus.cc:482] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:39:28.676043 19507 raft_consensus.cc:3037] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:28.677129 19507 raft_consensus.cc:504] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.677487 19507 leader_election.cc:288] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [CANDIDATE]: Term 1 election: Requested vote from peers f53064e78fd54ee88c6a33450ca30b5d (127.0.98.1:35399), 0dad4f3a43e342aa82dc9ba8143713a1 (127.0.98.2:34863)
I20260927 12:39:28.677886 19174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83315d30e8ab49bd8255fbb4491467a3" candidate_uuid: "174ef8933ded47aea5cce72788984cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53064e78fd54ee88c6a33450ca30b5d"
I20260927 12:39:28.677922 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83315d30e8ab49bd8255fbb4491467a3" candidate_uuid: "174ef8933ded47aea5cce72788984cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dad4f3a43e342aa82dc9ba8143713a1"
I20260927 12:39:28.678041 19305 raft_consensus.cc:3037] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:28.678043 19174 raft_consensus.cc:3037] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:28.679095 19305 raft_consensus.cc:2454] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174ef8933ded47aea5cce72788984cb2 in term 1.
I20260927 12:39:28.679255 19174 raft_consensus.cc:2454] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 174ef8933ded47aea5cce72788984cb2 in term 1.
I20260927 12:39:28.679443 19371 leader_election.cc:302] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dad4f3a43e342aa82dc9ba8143713a1, 174ef8933ded47aea5cce72788984cb2; no voters:
I20260927 12:39:28.679625 19507 raft_consensus.cc:2781] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:28.679831 19507 raft_consensus.cc:686] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 174ef8933ded47aea5cce72788984cb2, State: Running, Role: LEADER
I20260927 12:39:28.680230 19507 consensus_queue.cc:237] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } }
I20260927 12:39:28.683529 19032 catalog_manager.cc:5495] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 174ef8933ded47aea5cce72788984cb2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "174ef8933ded47aea5cce72788984cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174ef8933ded47aea5cce72788984cb2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 } health_report { overall_health: UNKNOWN } } }
W20260927 12:39:28.704017 19352 tablet.cc:2369] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:28.737378 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.733135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.98.3:43115)
0927 12:39:28.733268 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:28.733271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:28.733285 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:39:28.733397 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:28.733399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:28.733458 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:28.733552 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:39:28.733877 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.734405 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.735413 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:28.735744 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:28.735838 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:28.736327 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:39:28.736784 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:39:28.736901 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:28.737137 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 12:39:28.737183 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:39:28.743891 19305 raft_consensus.cc:1261] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 1 FOLLOWER]: Refusing update from remote peer 174ef8933ded47aea5cce72788984cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:28.743891 19174 raft_consensus.cc:1261] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 1 FOLLOWER]: Refusing update from remote peer 174ef8933ded47aea5cce72788984cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:28.744691 19509 consensus_queue.cc:1035] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53064e78fd54ee88c6a33450ca30b5d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:28.744880 19507 consensus_queue.cc:1035] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dad4f3a43e342aa82dc9ba8143713a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:28.753656 19513 mvcc.cc:205] Tried to move back new op lower bound from 7333940300767313920 to 7333940300517138432. Current Snapshot: MvccSnapshot[applied={T|T < 7333940300767313920}]
I20260927 12:39:28.762665 19028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:28.762842 19028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:28.765758 19028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5a01bb92b6a4b3286644844d77a90e6: Sending DeleteTablet for 3 replicas of tablet 83315d30e8ab49bd8255fbb4491467a3
I20260927 12:39:28.766712 19154 tablet_service.cc:1514] Processing DeleteTablet for tablet 83315d30e8ab49bd8255fbb4491467a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:28 UTC) from {username='slave'} at 127.0.0.1:56014
I20260927 12:39:28.767009 19285 tablet_service.cc:1514] Processing DeleteTablet for tablet 83315d30e8ab49bd8255fbb4491467a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:28 UTC) from {username='slave'} at 127.0.0.1:42256
I20260927 12:39:28.767432 19522 tablet_replica.cc:330] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d: stopping tablet replica
I20260927 12:39:28.767551 19416 tablet_service.cc:1514] Processing DeleteTablet for tablet 83315d30e8ab49bd8255fbb4491467a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:28 UTC) from {username='slave'} at 127.0.0.1:37392
I20260927 12:39:28.767976 19522 raft_consensus.cc:2229] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:28.768520 19522 raft_consensus.cc:2258] T 83315d30e8ab49bd8255fbb4491467a3 P f53064e78fd54ee88c6a33450ca30b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:28.768843 19524 tablet_replica.cc:330] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: stopping tablet replica
I20260927 12:39:28.769132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19092
I20260927 12:39:28.769167 19524 raft_consensus.cc:2229] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:39:28.770865 19524 raft_consensus.cc:2258] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:28.772296 19524 ts_tablet_manager.cc:1905] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:28.775350 19524 ts_tablet_manager.cc:1918] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:28.775489 19524 log.cc:1196] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/83315d30e8ab49bd8255fbb4491467a3
I20260927 12:39:28.775779 19524 ts_tablet_manager.cc:1939] T 83315d30e8ab49bd8255fbb4491467a3 P 174ef8933ded47aea5cce72788984cb2: Deleting consensus metadata
I20260927 12:39:28.777444 19523 tablet_replica.cc:330] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1: stopping tablet replica
I20260927 12:39:28.777408 19017 catalog_manager.cc:4841] TS 174ef8933ded47aea5cce72788984cb2 (127.0.98.3:43115): tablet 83315d30e8ab49bd8255fbb4491467a3 (table test-table [id=f693b94ce5764f0faacd5b76735e8a29]) successfully deleted
I20260927 12:39:28.777909 19523 raft_consensus.cc:2229] T 83315d30e8ab49bd8255fbb4491467a3 P 0dad4f3a43e342aa82dc9ba8143713a1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 12:39:28.778072 19017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35399 (error 108)
I20260927 12:39:28.778282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19223
I20260927 12:39:28.779873 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.779494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.0.1:56016)
0927 12:39:28.779691 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.779760 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260927 12:39:28.780105 19017 catalog_manager.cc:4584] TS f53064e78fd54ee88c6a33450ca30b5d (127.0.98.1:35399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83315d30e8ab49bd8255fbb4491467a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35399: connect: Connection refused (error 111)
I20260927 12:39:28.784142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19354
I20260927 12:39:28.785306 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:28.785005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34863 (local address 127.0.0.1:42262)
0927 12:39:28.785147 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:28.785211 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 12:39:28.790684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19001
2026-09-27T12:39:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1258 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:28 dist-test-slave-tzgx krb5kdc[19528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:28 dist-test-slave-tzgx krb5kdc[19528](info): set up 2 sockets
Sep 27 12:39:28 dist-test-slave-tzgx krb5kdc[19528](info): commencing operation
krb5kdc: starting...
W20260927 12:39:29.779984 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:29 dist-test-slave-tzgx krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:29.909603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:30.029778 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:30.030094 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:30.030136 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:30.034483 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:30.034569 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:30.034590 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:30.034605 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:30.034618 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:30.040747 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:30.042098 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:30.043243 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:30.052340 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.052320 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.052320 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:30.052382 19541 server_base.cc:1034] running on GCE node
I20260927 12:39:30.053402 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:30.054641 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:30.055840 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790512770055823 us; error 35 us; skew 500 ppm
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:30.059113 19541 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:30.060262 19541 webserver.cc:460] Webserver started at http://127.0.98.62:45973/ using document root <none> and password file <none>
I20260927 12:39:30.060863 19541 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:30.060940 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:30.061161 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:30.063351 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "0721f97263664a9085653320cdcfc636"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "4282b6dcd0a4ad263b7f5c5cc3c62654"
server_key_iv: "a9138cc353aed7d5248b6b1dcbb2c059"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.063974 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "0721f97263664a9085653320cdcfc636"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "4282b6dcd0a4ad263b7f5c5cc3c62654"
server_key_iv: "a9138cc353aed7d5248b6b1dcbb2c059"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.068222 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:30.071198 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.073226 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:30.073400 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "0721f97263664a9085653320cdcfc636"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "4282b6dcd0a4ad263b7f5c5cc3c62654"
server_key_iv: "a9138cc353aed7d5248b6b1dcbb2c059"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.073539 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:30.087715 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:30.091154 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:30.091380 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:30.098938 19541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44555
I20260927 12:39:30.099000 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44555 every 8 connection(s)
I20260927 12:39:30.099975 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:30.102913 19609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:30.106688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19541
I20260927 12:39:30.106820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:30.107086 392 external_mini_cluster.cc:1428] Setting key 68a89cf6fa8e870c11557676e9ec0c7e
I20260927 12:39:30.109411 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636: Bootstrap starting.
I20260927 12:39:30.112205 19609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:30.112993 19609 log.cc:824] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:30.115535 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636: No bootstrap required, opened a new log
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:30.119719 19609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0721f97263664a9085653320cdcfc636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260927 12:39:30.120048 19609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:30.120136 19609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0721f97263664a9085653320cdcfc636, State: Initialized, Role: FOLLOWER
I20260927 12:39:30.120646 19609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0721f97263664a9085653320cdcfc636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260927 12:39:30.120786 19609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:30.120858 19609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:30.120958 19609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:30.121991 19609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0721f97263664a9085653320cdcfc636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260927 12:39:30.122326 19609 leader_election.cc:302] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0721f97263664a9085653320cdcfc636; no voters:
I20260927 12:39:30.122414 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.108930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.62:44555)
0927 12:39:30.109418 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.109435 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.109473 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:39:30.110187 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.110206 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.110609 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.110920 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:30.111918 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.113085 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.114100 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.114631 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.114751 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.117686 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:30.117692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:30.117707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:30.117740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:30.119406 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.119944 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.119949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.119954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.120069 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.120390 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.120394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.120396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.120860 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:30.121057 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.121323 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.121626 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":149,"threads_started":1}
I20260927 12:39:30.122665 19609 leader_election.cc:288] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:30.122757 19614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:30.122951 19614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [term 1 LEADER]: Becoming Leader. State: Replica: 0721f97263664a9085653320cdcfc636, State: Running, Role: LEADER
I20260927 12:39:30.123255 19614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0721f97263664a9085653320cdcfc636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260927 12:39:30.123926 19609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:30.124853 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0721f97263664a9085653320cdcfc636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0721f97263664a9085653320cdcfc636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } }
I20260927 12:39:30.124991 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:30.125289 19616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0721f97263664a9085653320cdcfc636. Latest consensus state: current_term: 1 leader_uuid: "0721f97263664a9085653320cdcfc636" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0721f97263664a9085653320cdcfc636" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } }
I20260927 12:39:30.125386 19616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:30.125540 19623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:30.128387 19623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:30.134208 19623 catalog_manager.cc:1347] Generated new cluster ID: 9a12e79f061044c18d2cc7c0190f7d9e
I20260927 12:39:30.134296 19623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:30.147861 19623 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:30.149103 19623 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:30.160199 19623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636: Generated new TSK 0
I20260927 12:39:30.161022 19623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:39:30.197173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:30.330561 19635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:30.330869 19635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:30.330914 19635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:30.335247 19635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:30.335323 19635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:30.335388 19635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:30.340979 19635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:30.342128 19635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:30.343173 19635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:30.353209 19640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.353201 19641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.353209 19643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:30.354287 19635 server_base.cc:1034] running on GCE node
I20260927 12:39:30.354794 19635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:30.355526 19635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:30.356695 19635 hybrid_clock.cc:648] HybridClock initialized: now 1790512770356673 us; error 37 us; skew 500 ppm
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:30.359790 19635 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:39:30.361028 19635 webserver.cc:460] Webserver started at http://127.0.98.1:36269/ using document root <none> and password file <none>
I20260927 12:39:30.361663 19635 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:30.361739 19635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:30.361948 19635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:30.364283 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "7045754af63f4ae8ba971dc8eeab735d"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "79334856fe2e05d12d6798a3ab41021a"
server_key_iv: "d341850d6be0afcef09b64dd79db92ea"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.364923 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7045754af63f4ae8ba971dc8eeab735d"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "79334856fe2e05d12d6798a3ab41021a"
server_key_iv: "d341850d6be0afcef09b64dd79db92ea"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.369239 19635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:30.372025 19650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.373473 19635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:30.373607 19635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "7045754af63f4ae8ba971dc8eeab735d"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "79334856fe2e05d12d6798a3ab41021a"
server_key_iv: "d341850d6be0afcef09b64dd79db92ea"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.373728 19635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:30.386587 19635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:30.389802 19635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:30.389997 19635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:30.390579 19635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:30.391392 19635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:30.391439 19635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.391491 19635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:30.391516 19635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.400868 19635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35223
I20260927 12:39:30.400892 19763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35223 every 8 connection(s)
I20260927 12:39:30.401823 19635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:30.404364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19635
I20260927 12:39:30.404510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:39:30.404780 392 external_mini_cluster.cc:1428] Setting key 5319627cd4042ffb074db289816b2830
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:39:30.417773 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.403916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50979 (local address 127.0.98.62:44555)
0927 12:39:30.404064 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.404068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.405408 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 12:39:30.406310 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.406313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.406368 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.406454 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:30.407697 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.408314 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.410086 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.410371 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.410480 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.413580 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:30.413585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:30.413588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:30.413625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:30.415132 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.415787 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.415791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.415793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.415894 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.416324 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.416361 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.416362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.416588 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:30.416681 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.417461 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.417599 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:39:30.418669 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.404359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:50979)
0927 12:39:30.405242 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.405302 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.406109 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.406593 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.406600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.406966 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:30.407518 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.407540 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.408514 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.408518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.409918 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.409932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.410581 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.410585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.410822 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.411484 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:30.411502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:30.413325 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:30.415283 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:30.415289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:30.415307 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:30.415637 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:30.416008 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:30.416011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:30.416012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:30.416210 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:30.416718 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:30.416729 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:30.417130 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.417314 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.417706 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":144,"threads_started":1}
I20260927 12:39:30.420274 19764 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260927 12:39:30.420532 19764 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:30.421077 19764 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260927 12:39:30.422711 19573 ts_manager.cc:194] Registered new tserver with Master: 7045754af63f4ae8ba971dc8eeab735d (127.0.98.1:35223)
I20260927 12:39:30.423982 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:39:30.436437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:30.551314 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:30.551585 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:30.551630 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:30.555847 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:30.555933 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:30.555995 19769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:30.561465 19769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:30.562553 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:30.563548 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:39:30.573728 19769 server_base.cc:1034] running on GCE node
W20260927 12:39:30.573696 19774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.573803 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.573684 19775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:30.574846 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:30.575660 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:30.576858 19769 hybrid_clock.cc:648] HybridClock initialized: now 1790512770576840 us; error 34 us; skew 500 ppm
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:30.579918 19769 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:39:30.580957 19769 webserver.cc:460] Webserver started at http://127.0.98.2:45595/ using document root <none> and password file <none>
I20260927 12:39:30.581524 19769 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:30.581596 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:30.581789 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:30.584465 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "7fce33b1ea1e448ba77a0f9bca0be79f"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "ac43aaf1423928cbe2c8ed2f76c76af1"
server_key_iv: "c115c234bf657d36b9a03f0c4583fd7b"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.585103 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7fce33b1ea1e448ba77a0f9bca0be79f"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "ac43aaf1423928cbe2c8ed2f76c76af1"
server_key_iv: "c115c234bf657d36b9a03f0c4583fd7b"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.589082 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 12:39:30.591696 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.593129 19769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:30.593262 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "7fce33b1ea1e448ba77a0f9bca0be79f"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "ac43aaf1423928cbe2c8ed2f76c76af1"
server_key_iv: "c115c234bf657d36b9a03f0c4583fd7b"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.593384 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:30.607161 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:30.610394 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:30.610584 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:30.611174 19769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:30.612021 19769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:30.612067 19769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.612152 19769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:30.612179 19769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.621318 19769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33615
I20260927 12:39:30.621397 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33615 every 8 connection(s)
I20260927 12:39:30.622280 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:30.623202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19769
I20260927 12:39:30.623309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:39:30.623567 392 external_mini_cluster.cc:1428] Setting key 866980db681302e1c8e2c7055ced40db
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:39:30.639418 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.624849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48605 (local address 127.0.98.62:44555)
0927 12:39:30.624981 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.624986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.625784 (+ 798us) server_negotiation.cc:407] Connection header received
0927 12:39:30.627058 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.627062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.627122 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.627214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:30.628929 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.629685 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.631761 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.632020 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.632203 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.635528 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:30.635532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:30.635534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:30.635564 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:30.637041 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.637611 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.637615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.637617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.637738 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.638112 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.638116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.638118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.638347 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:30.638461 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.639059 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.639191 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:39:30.640560 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.624868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:48605)
0927 12:39:30.625595 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.625664 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.626759 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.627429 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.627437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.627809 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:30.628701 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.628736 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.629852 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.629856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.631573 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.631591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.632277 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.632282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.632578 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.633335 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:30.633442 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:30.635296 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:30.637182 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:30.637190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:30.637212 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:30.637504 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:30.637846 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:30.637849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:30.637851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:30.638025 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:30.638483 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:30.638489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:30.638775 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.638994 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.639382 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":158,"threads_started":1}
I20260927 12:39:30.641914 19898 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260927 12:39:30.642171 19898 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:30.642737 19898 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260927 12:39:30.643935 19573 ts_manager.cc:194] Registered new tserver with Master: 7fce33b1ea1e448ba77a0f9bca0be79f (127.0.98.2:33615)
I20260927 12:39:30.644552 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:39:30.655993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:30.768194 19903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:30.768496 19903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:30.768543 19903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:30.772763 19903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:30.772832 19903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:30.772895 19903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:30.778833 19903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:30.779920 19903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:30.780936 19903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:30.791299 19911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.791287 19908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:30.791312 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:30.791452 19903 server_base.cc:1034] running on GCE node
I20260927 12:39:30.792013 19903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:30.792733 19903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:30.793921 19903 hybrid_clock.cc:648] HybridClock initialized: now 1790512770793900 us; error 30 us; skew 500 ppm
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:30.796845 19903 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:39:30.797827 19903 webserver.cc:460] Webserver started at http://127.0.98.3:42765/ using document root <none> and password file <none>
I20260927 12:39:30.798411 19903 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:30.798475 19903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:30.798669 19903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:30.800848 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "26b65df38c3a48a0b069f4eb53762f3c"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "3631af4db2d82d46fbb16b0b371bf752"
server_key_iv: "899851fd7a845a00d6c30aad7e02b86a"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.801421 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "26b65df38c3a48a0b069f4eb53762f3c"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "3631af4db2d82d46fbb16b0b371bf752"
server_key_iv: "899851fd7a845a00d6c30aad7e02b86a"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.805341 19903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:30.808033 19918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.809440 19903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:30.809574 19903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "26b65df38c3a48a0b069f4eb53762f3c"
format_stamp: "Formatted at 2026-09-27 12:39:30 on dist-test-slave-tzgx"
server_key: "3631af4db2d82d46fbb16b0b371bf752"
server_key_iv: "899851fd7a845a00d6c30aad7e02b86a"
server_key_version: "encryptionkey@0"
I20260927 12:39:30.809715 19903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:30.827072 19903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:30.830456 19903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:30.830628 19903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:30.831243 19903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:30.832145 19903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:30.832191 19903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.832233 19903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:30.832247 19903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:30.841248 19903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39787
I20260927 12:39:30.841272 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39787 every 8 connection(s)
I20260927 12:39:30.842186 19903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:30.842819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 19903
I20260927 12:39:30.842960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:39:30.843339 392 external_mini_cluster.cc:1428] Setting key 1c1b856798f2076cd19b41211d31dd78
Sep 27 12:39:30 dist-test-slave-tzgx krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:30.857506 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.844434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47285 (local address 127.0.98.62:44555)
0927 12:39:30.844562 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.844567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.846007 (+ 1440us) server_negotiation.cc:407] Connection header received
0927 12:39:30.846977 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.846980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.847037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.847109 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:30.848428 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.848974 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.850187 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.850414 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.850501 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.853202 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:30.853206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:30.853208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:30.853239 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:30.854987 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.855618 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.855622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.855624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.855707 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.856085 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.856108 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.856110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.856318 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:30.856406 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.857207 (+ 801us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.857348 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:39:30.858443 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.844935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:47285)
0927 12:39:30.845846 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.845914 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.846762 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.847288 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.847296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.847697 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:30.848273 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.848291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.849111 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.849114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.850054 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.850066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.850576 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.850580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.850748 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.851331 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:30.851349 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:30.852990 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:30.855165 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:30.855172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:30.855189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:30.855508 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:30.855812 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:30.855814 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:30.855816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:30.855972 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:30.856604 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:30.856611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:30.856962 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.857159 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.857524 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":174,"threads_started":1}
I20260927 12:39:30.859807 20032 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260927 12:39:30.860037 20032 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:30.860566 20032 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260927 12:39:30.861822 19573 ts_manager.cc:194] Registered new tserver with Master: 26b65df38c3a48a0b069f4eb53762f3c (127.0.98.3:39787)
I20260927 12:39:30.862390 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47285
I20260927 12:39:30.870206 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:39:30.880260 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.872287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.0.98.62:44555)
0927 12:39:30.872504 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.872509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.872636 (+ 127us) server_negotiation.cc:407] Connection header received
0927 12:39:30.872786 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.872790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.872833 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.872900 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:30.873749 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.874274 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.875128 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.875345 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.875435 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.876801 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:30.876805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:30.876807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:30.876834 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:30.878406 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.878871 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.878875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.878877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.878966 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:30.879239 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:30.879243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:30.879245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:30.879498 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:30.879631 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.879923 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.880045 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 12:39:30.883328 19573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:39:30.886524 19573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:39:30.902051 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.896430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.0.98.3:39787)
0927 12:39:30.896913 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.896947 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.896973 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:39:30.897148 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.897153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.897356 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.897513 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:30.897844 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.899258 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.900767 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.901318 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.901420 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.901592 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.901769 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.901870 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":131,"threads_started":1}
I20260927 12:39:30.902050 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.895618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39787 (local address 127.0.0.1:52294)
0927 12:39:30.896885 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.896897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.897012 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.897511 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.897519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.897540 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:30.897682 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.897690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.899383 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.899387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.900643 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.900651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.901458 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.901462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.901574 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.901612 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.901727 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.901871 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":10,"spinlock_wait_cycles":17792,"thread_start_us":101,"threads_started":1}
I20260927 12:39:30.902108 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.895752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33615 (local address 127.0.0.1:48116)
0927 12:39:30.897169 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.897182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.897281 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.897884 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.897886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.897900 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:30.897984 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.897991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.898992 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.898996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.900340 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.900351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.901670 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.901674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.901760 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.901784 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.901849 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.901951 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260927 12:39:30.902549 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.896470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.0.98.2:33615)
0927 12:39:30.897036 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.897047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.897282 (+ 235us) server_negotiation.cc:407] Connection header received
0927 12:39:30.897391 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.897396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.897573 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.897704 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:30.898112 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.898849 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.900515 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.901344 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.901456 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.901604 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.902301 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.902405 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":111,"threads_started":1}
I20260927 12:39:30.902578 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.895586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35223 (local address 127.0.0.1:56734)
0927 12:39:30.896395 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.896452 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.896592 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.897764 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.897767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.897783 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:30.897886 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.897893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.899287 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.899290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.900596 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.900608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.901781 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.901784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.901863 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.901881 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.901936 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.901991 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":35,"thread_start_us":90,"threads_started":1}
I20260927 12:39:30.903047 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.895607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.1:35223)
0927 12:39:30.897167 (+ 1560us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.897180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.897204 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:39:30.897280 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.897285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.897465 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.897606 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:30.898004 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.899129 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.900734 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.901510 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.901602 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.901730 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.902114 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.902872 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":114,"threads_started":1}
I20260927 12:39:30.905285 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 6e157ffc15374aec8616b4c624fc279b (DEFAULT_TABLE table=test-table [id=b499212d61dd4dbab616dddc714a4c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:30.905393 19966 tablet_service.cc:1467] Processing CreateTablet for tablet 6e157ffc15374aec8616b4c624fc279b (DEFAULT_TABLE table=test-table [id=b499212d61dd4dbab616dddc714a4c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:30.906342 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e157ffc15374aec8616b4c624fc279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:30.905994 19698 tablet_service.cc:1467] Processing CreateTablet for tablet 6e157ffc15374aec8616b4c624fc279b (DEFAULT_TABLE table=test-table [id=b499212d61dd4dbab616dddc714a4c80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:30.906448 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e157ffc15374aec8616b4c624fc279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:30.906982 19698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e157ffc15374aec8616b4c624fc279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:30.913821 20050 tablet_bootstrap.cc:492] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Bootstrap starting.
I20260927 12:39:30.914330 20049 tablet_bootstrap.cc:492] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Bootstrap starting.
I20260927 12:39:30.916532 20051 tablet_bootstrap.cc:492] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: Bootstrap starting.
I20260927 12:39:30.916720 20049 tablet_bootstrap.cc:654] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:30.916862 20050 tablet_bootstrap.cc:654] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:30.918131 20049 log.cc:824] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:30.918131 20050 log.cc:824] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:30.918715 20051 tablet_bootstrap.cc:654] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:30.919926 20051 log.cc:824] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:30.920506 20050 tablet_bootstrap.cc:492] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: No bootstrap required, opened a new log
I20260927 12:39:30.920506 20049 tablet_bootstrap.cc:492] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: No bootstrap required, opened a new log
I20260927 12:39:30.920723 20050 ts_tablet_manager.cc:1397] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:30.920723 20049 ts_tablet_manager.cc:1397] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:30.921718 20051 tablet_bootstrap.cc:492] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: No bootstrap required, opened a new log
I20260927 12:39:30.921916 20051 ts_tablet_manager.cc:1397] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:39:30.925168 20051 raft_consensus.cc:348] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.925187 20049 raft_consensus.cc:348] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.925486 20051 raft_consensus.cc:374] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:30.925493 20049 raft_consensus.cc:374] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:30.925328 20050 raft_consensus.cc:348] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.925546 20051 raft_consensus.cc:729] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7045754af63f4ae8ba971dc8eeab735d, State: Initialized, Role: FOLLOWER
I20260927 12:39:30.925547 20049 raft_consensus.cc:729] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fce33b1ea1e448ba77a0f9bca0be79f, State: Initialized, Role: FOLLOWER
I20260927 12:39:30.925563 20050 raft_consensus.cc:374] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:30.925604 20050 raft_consensus.cc:729] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b65df38c3a48a0b069f4eb53762f3c, State: Initialized, Role: FOLLOWER
I20260927 12:39:30.926077 20051 consensus_queue.cc:260] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.926081 20050 consensus_queue.cc:260] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.926081 20049 consensus_queue.cc:260] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.926745 19764 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260927 12:39:30.927196 20051 ts_tablet_manager.cc:1428] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:39:30.927198 20050 ts_tablet_manager.cc:1428] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 12:39:30.927197 20049 ts_tablet_manager.cc:1428] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:39:30.927549 19898 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260927 12:39:30.928260 20032 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260927 12:39:30.945266 20055 raft_consensus.cc:482] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:30.945483 20055 raft_consensus.cc:504] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.946583 20055 leader_election.cc:288] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26b65df38c3a48a0b069f4eb53762f3c (127.0.98.3:39787), 7fce33b1ea1e448ba77a0f9bca0be79f (127.0.98.2:33615)
I20260927 12:39:30.950914 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.946982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57859 (local address 127.0.98.2:33615)
0927 12:39:30.947118 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.947123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.947344 (+ 221us) server_negotiation.cc:407] Connection header received
0927 12:39:30.947456 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.947460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.947508 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.947576 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:30.947859 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.948447 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.949800 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.950347 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.950442 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.950502 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.950720 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.950791 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:39:30.950933 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.946866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33615 (local address 127.0.98.1:57859)
0927 12:39:30.947247 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.947259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.947359 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.947593 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.947596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.947612 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:30.947750 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.947757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.948549 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.948553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.949651 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.949661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.950463 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.950467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.950557 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.950609 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.950678 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.950769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":94,"thread_start_us":137,"threads_started":1}
I20260927 12:39:30.950910 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.946988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51233 (local address 127.0.98.3:39787)
0927 12:39:30.947117 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.947122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.947451 (+ 329us) server_negotiation.cc:407] Connection header received
0927 12:39:30.947551 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.947554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.947605 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.947678 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:30.947926 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.948499 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.949826 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.950309 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.950376 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.950428 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.950703 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.950783 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:39:30.950948 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.946872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39787 (local address 127.0.98.1:51233)
0927 12:39:30.947350 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:30.947363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:30.947469 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:30.947697 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:30.947700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:30.947715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:30.947823 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.947830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.948613 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.948616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.949684 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:30.949693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.950502 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.950504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:30.950597 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.950613 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:39:30.950664 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 12:39:30.950772 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":27,"thread_start_us":109,"threads_started":1}
I20260927 12:39:30.951671 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e157ffc15374aec8616b4c624fc279b" candidate_uuid: "7045754af63f4ae8ba971dc8eeab735d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" is_pre_election: true
I20260927 12:39:30.952008 19852 raft_consensus.cc:2454] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7045754af63f4ae8ba971dc8eeab735d in term 0.
I20260927 12:39:30.952198 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e157ffc15374aec8616b4c624fc279b" candidate_uuid: "7045754af63f4ae8ba971dc8eeab735d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b65df38c3a48a0b069f4eb53762f3c" is_pre_election: true
I20260927 12:39:30.952476 19986 raft_consensus.cc:2454] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7045754af63f4ae8ba971dc8eeab735d in term 0.
I20260927 12:39:30.952505 19654 leader_election.cc:302] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7045754af63f4ae8ba971dc8eeab735d, 7fce33b1ea1e448ba77a0f9bca0be79f; no voters:
I20260927 12:39:30.952770 20055 raft_consensus.cc:2781] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:39:30.952867 20055 raft_consensus.cc:482] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:39:30.952939 20055 raft_consensus.cc:3037] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:30.953961 20055 raft_consensus.cc:504] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.954295 20055 leader_election.cc:288] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [CANDIDATE]: Term 1 election: Requested vote from peers 26b65df38c3a48a0b069f4eb53762f3c (127.0.98.3:39787), 7fce33b1ea1e448ba77a0f9bca0be79f (127.0.98.2:33615)
I20260927 12:39:30.954720 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e157ffc15374aec8616b4c624fc279b" candidate_uuid: "7045754af63f4ae8ba971dc8eeab735d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b65df38c3a48a0b069f4eb53762f3c"
I20260927 12:39:30.954761 19852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e157ffc15374aec8616b4c624fc279b" candidate_uuid: "7045754af63f4ae8ba971dc8eeab735d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f"
I20260927 12:39:30.954834 19986 raft_consensus.cc:3037] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:30.954880 19852 raft_consensus.cc:3037] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:30.955915 19852 raft_consensus.cc:2454] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7045754af63f4ae8ba971dc8eeab735d in term 1.
I20260927 12:39:30.955925 19986 raft_consensus.cc:2454] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7045754af63f4ae8ba971dc8eeab735d in term 1.
I20260927 12:39:30.956246 19653 leader_election.cc:302] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26b65df38c3a48a0b069f4eb53762f3c, 7045754af63f4ae8ba971dc8eeab735d; no voters:
I20260927 12:39:30.956432 20055 raft_consensus.cc:2781] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:30.956645 20055 raft_consensus.cc:686] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 1 LEADER]: Becoming Leader. State: Replica: 7045754af63f4ae8ba971dc8eeab735d, State: Running, Role: LEADER
I20260927 12:39:30.956971 20055 consensus_queue.cc:237] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } }
I20260927 12:39:30.960374 19573 catalog_manager.cc:5495] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7045754af63f4ae8ba971dc8eeab735d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7045754af63f4ae8ba971dc8eeab735d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7045754af63f4ae8ba971dc8eeab735d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 } health_report { overall_health: UNKNOWN } } }
I20260927 12:39:30.986964 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:30.983345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.0.98.1:35223)
0927 12:39:30.983508 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:30.983512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:30.983526 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:39:30.983610 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:30.983612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:30.983666 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:30.983759 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:39:30.984010 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.984595 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.985504 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:30.985715 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:30.985794 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:30.986120 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:39:30.986572 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:39:30.986660 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:30.986763 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 12:39:30.986820 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 12:39:30.993250 19852 raft_consensus.cc:1261] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 1 FOLLOWER]: Refusing update from remote peer 7045754af63f4ae8ba971dc8eeab735d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:30.993249 19986 raft_consensus.cc:1261] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 1 FOLLOWER]: Refusing update from remote peer 7045754af63f4ae8ba971dc8eeab735d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:30.994007 20060 consensus_queue.cc:1035] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b65df38c3a48a0b069f4eb53762f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:30.994204 20055 consensus_queue.cc:1035] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fce33b1ea1e448ba77a0f9bca0be79f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:31.001842 20063 mvcc.cc:205] Tried to move back new op lower bound from 7333940309979987968 to 7333940309842460672. Current Snapshot: MvccSnapshot[applied={T|T < 7333940309979987968}]
I20260927 12:39:31.002529 20064 mvcc.cc:205] Tried to move back new op lower bound from 7333940309979987968 to 7333940309842460672. Current Snapshot: MvccSnapshot[applied={T|T < 7333940309979987968}]
I20260927 12:39:31.003099 20065 mvcc.cc:205] Tried to move back new op lower bound from 7333940309979987968 to 7333940309842460672. Current Snapshot: MvccSnapshot[applied={T|T < 7333940309979987968}]
I20260927 12:39:31.011559 19567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:31.011739 19567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:31.014545 19567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0721f97263664a9085653320cdcfc636: Sending DeleteTablet for 3 replicas of tablet 6e157ffc15374aec8616b4c624fc279b
I20260927 12:39:31.015482 19698 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e157ffc15374aec8616b4c624fc279b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56734
I20260927 12:39:31.015530 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e157ffc15374aec8616b4c624fc279b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48116
I20260927 12:39:31.016062 20073 tablet_replica.cc:330] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: stopping tablet replica
I20260927 12:39:31.016062 20074 tablet_replica.cc:330] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d: stopping tablet replica
I20260927 12:39:31.016125 19966 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e157ffc15374aec8616b4c624fc279b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52294
I20260927 12:39:31.016439 20073 raft_consensus.cc:2229] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:31.016481 20074 raft_consensus.cc:2229] T 6e157ffc15374aec8616b4c624fc279b P 7045754af63f4ae8ba971dc8eeab735d [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:39:31.016814 20073 raft_consensus.cc:2258] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:31.017098 20075 tablet_replica.cc:330] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: stopping tablet replica
I20260927 12:39:31.017397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19635
I20260927 12:39:31.017395 20075 raft_consensus.cc:2229] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:31.018698 20073 ts_tablet_manager.cc:1905] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:31.019155 20075 raft_consensus.cc:2258] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:31.020934 20075 ts_tablet_manager.cc:1905] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:31.021399 20073 ts_tablet_manager.cc:1918] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:31.021489 20073 log.cc:1196] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/6e157ffc15374aec8616b4c624fc279b
I20260927 12:39:31.021757 20073 ts_tablet_manager.cc:1939] T 6e157ffc15374aec8616b4c624fc279b P 7fce33b1ea1e448ba77a0f9bca0be79f: Deleting consensus metadata
I20260927 12:39:31.022958 19560 catalog_manager.cc:4841] TS 7fce33b1ea1e448ba77a0f9bca0be79f (127.0.98.2:33615): tablet 6e157ffc15374aec8616b4c624fc279b (table test-table [id=b499212d61dd4dbab616dddc714a4c80]) successfully deleted
I20260927 12:39:31.023588 20075 ts_tablet_manager.cc:1918] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:31.023746 20075 log.cc:1196] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/6e157ffc15374aec8616b4c624fc279b
I20260927 12:39:31.024180 20075 ts_tablet_manager.cc:1939] T 6e157ffc15374aec8616b4c624fc279b P 26b65df38c3a48a0b069f4eb53762f3c: Deleting consensus metadata
W20260927 12:39:31.025378 19558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35223 (error 108)
I20260927 12:39:31.025717 19559 catalog_manager.cc:4841] TS 26b65df38c3a48a0b069f4eb53762f3c (127.0.98.3:39787): tablet 6e157ffc15374aec8616b4c624fc279b (table test-table [id=b499212d61dd4dbab616dddc714a4c80]) successfully deleted
I20260927 12:39:31.026149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19769
I20260927 12:39:31.026428 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.026160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35223 (local address 127.0.0.1:56754)
0927 12:39:31.026286 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.026346 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 12:39:31.026899 19558 catalog_manager.cc:4584] TS 7045754af63f4ae8ba971dc8eeab735d (127.0.98.1:35223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e157ffc15374aec8616b4c624fc279b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35223: connect: Connection refused (error 111)
I20260927 12:39:31.031239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19903
I20260927 12:39:31.036500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 19541
2026-09-27T12:39:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T12:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:31Z Disabled control of system clock
I20260927 12:39:31.066394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:39:31.183210 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:31.183514 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:31.183552 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:31.187752 20080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:31.187839 20080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:31.187853 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:31.187870 20080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:31.187883 20080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:31.193926 20080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:31.195037 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:31.196467 20080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:31.204952 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.204970 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.204950 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:31.205420 20080 server_base.cc:1034] running on GCE node
I20260927 12:39:31.206079 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:31.207283 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:31.208501 20080 hybrid_clock.cc:648] HybridClock initialized: now 1790512771208476 us; error 48 us; skew 500 ppm
I20260927 12:39:31.210482 20080 webserver.cc:460] Webserver started at http://127.0.98.62:33687/ using document root <none> and password file <none>
I20260927 12:39:31.211061 20080 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:31.211134 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:31.211330 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:31.213560 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "5773b3f288454aed8090a06ddfa58623"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "b24a4cbd7b3568162e7d436f4b13574e"
server_key_iv: "6a093cb5ef2c9d891a5d7cde56e13bc3"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.214123 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "5773b3f288454aed8090a06ddfa58623"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "b24a4cbd7b3568162e7d436f4b13574e"
server_key_iv: "6a093cb5ef2c9d891a5d7cde56e13bc3"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.218012 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:31.220631 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.221978 20080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:31.222114 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "5773b3f288454aed8090a06ddfa58623"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "b24a4cbd7b3568162e7d436f4b13574e"
server_key_iv: "6a093cb5ef2c9d891a5d7cde56e13bc3"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.222229 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:31.263033 20080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:31.263741 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:31.263911 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:31.270517 20080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37431
I20260927 12:39:31.270533 20146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37431 every 8 connection(s)
I20260927 12:39:31.271471 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:31.273252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20080
I20260927 12:39:31.273347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:31.273618 392 external_mini_cluster.cc:1428] Setting key 98606697511f423c0457694561397d64
I20260927 12:39:31.274945 20147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:31.281272 20147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623: Bootstrap starting.
I20260927 12:39:31.282981 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.274931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.0.98.62:37431)
0927 12:39:31.275600 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.275618 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.275656 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:39:31.276375 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.276393 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.276800 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.277096 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:31.277439 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.278411 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.279602 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.280134 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.280302 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.281112 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:31.281117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:31.281130 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:31.281592 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:31.281651 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:31.281672 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:31.281748 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.281944 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.282235 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260927 12:39:31.283977 20147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:31.284893 20147 log.cc:824] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:31.286795 20147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623: No bootstrap required, opened a new log
I20260927 12:39:31.290381 20147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773b3f288454aed8090a06ddfa58623" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 12:39:31.290618 20147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:31.290659 20147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5773b3f288454aed8090a06ddfa58623, State: Initialized, Role: FOLLOWER
I20260927 12:39:31.291026 20147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773b3f288454aed8090a06ddfa58623" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 12:39:31.291137 20147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:31.291183 20147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:31.291252 20147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:31.292213 20147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773b3f288454aed8090a06ddfa58623" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 12:39:31.292538 20147 leader_election.cc:302] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5773b3f288454aed8090a06ddfa58623; no voters:
I20260927 12:39:31.292798 20147 leader_election.cc:288] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:31.292906 20152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:31.293135 20152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [term 1 LEADER]: Becoming Leader. State: Replica: 5773b3f288454aed8090a06ddfa58623, State: Running, Role: LEADER
I20260927 12:39:31.293473 20152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773b3f288454aed8090a06ddfa58623" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 12:39:31.293967 20147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:31.295153 20153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5773b3f288454aed8090a06ddfa58623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773b3f288454aed8090a06ddfa58623" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260927 12:39:31.295193 20154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5773b3f288454aed8090a06ddfa58623. Latest consensus state: current_term: 1 leader_uuid: "5773b3f288454aed8090a06ddfa58623" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773b3f288454aed8090a06ddfa58623" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260927 12:39:31.295295 20153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:31.295296 20154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:31.295684 20161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:31.298846 20161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:31.304529 20161 catalog_manager.cc:1347] Generated new cluster ID: a4920efe2d7a4bc1b5b39dc15f83679a
I20260927 12:39:31.304615 20161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:31.314111 20161 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:31.315061 20161 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:31.325305 20161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623: Generated new TSK 0
I20260927 12:39:31.326146 20161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:31.338773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--builtin_ntp_servers=127.0.98.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:39:31.456084 20171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:31.456437 20171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:31.456496 20171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:31.460703 20171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:31.460776 20171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:31.460856 20171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:39:31.465961 20171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:31.467127 20171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:31.468828 20171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:31.479077 20177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.479140 20176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.479079 20179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:31.479414 20171 server_base.cc:1034] running on GCE node
I20260927 12:39:31.480108 20171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:31.480851 20171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:31.482034 20171 hybrid_clock.cc:648] HybridClock initialized: now 1790512771482019 us; error 28 us; skew 500 ppm
I20260927 12:39:31.485476 20171 webserver.cc:460] Webserver started at http://127.0.98.1:44087/ using document root <none> and password file <none>
I20260927 12:39:31.486100 20171 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:31.486172 20171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:31.486371 20171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:31.488554 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f5eb99b68eb4e3a894878d655fefaf9"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "24033987c48722c8750bcbf77c11425e"
server_key_iv: "c792c51c8188aadeddbe2d23b2066ea9"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.489133 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f5eb99b68eb4e3a894878d655fefaf9"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "24033987c48722c8750bcbf77c11425e"
server_key_iv: "c792c51c8188aadeddbe2d23b2066ea9"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.493635 20171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:39:31.496851 20185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.498410 20171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:39:31.498564 20171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal
uuid: "8f5eb99b68eb4e3a894878d655fefaf9"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "24033987c48722c8750bcbf77c11425e"
server_key_iv: "c792c51c8188aadeddbe2d23b2066ea9"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.498696 20171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:31.520359 20171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:31.521135 20171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:31.521333 20171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:31.522030 20171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:31.523030 20171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:31.523078 20171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.523132 20171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:31.523160 20171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.531968 20171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43875
I20260927 12:39:31.531999 20298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43875 every 8 connection(s)
I20260927 12:39:31.532949 20171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:39:31.535806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20171
I20260927 12:39:31.535939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:39:31.536253 392 external_mini_cluster.cc:1428] Setting key 0e2913adeead08e25f21e1dd563b6874
I20260927 12:39:31.538213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--builtin_ntp_servers=127.0.98.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:39:31.543202 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.534975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56777 (local address 127.0.98.62:37431)
0927 12:39:31.535156 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.535160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.536434 (+ 1274us) server_negotiation.cc:407] Connection header received
0927 12:39:31.537971 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.537974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.538032 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.538102 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:31.539307 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.539955 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.541188 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.541417 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.541505 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.542701 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:31.542706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:31.542709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:31.542724 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:31.542746 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:31.542754 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:31.542813 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.542965 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.543047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 12:39:31.544134 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.535466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:56777)
0927 12:39:31.536272 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.536328 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:31.537759 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:31.538288 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:31.538295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:31.538405 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:31.539123 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.539146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.540124 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.540128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.541048 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.541059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.541680 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.541683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.541837 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.542421 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:31.542423 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:31.542466 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:31.542540 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:31.542824 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:31.542833 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:31.542840 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:39:31.542957 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 12:39:31.543273 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260927 12:39:31.545786 20299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 12:39:31.546043 20299 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:31.546629 20299 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 12:39:31.548352 20110 ts_manager.cc:194] Registered new tserver with Master: 8f5eb99b68eb4e3a894878d655fefaf9 (127.0.98.1:43875)
I20260927 12:39:31.549544 20110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56777
W20260927 12:39:31.658192 20302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:31.658527 20302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:31.658568 20302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:31.662981 20302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:31.663064 20302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:31.663122 20302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:39:31.668390 20302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:31.669570 20302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:31.671017 20302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:31.681272 20310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.681313 20307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.681473 20308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:31.681499 20302 server_base.cc:1034] running on GCE node
I20260927 12:39:31.682052 20302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:31.682729 20302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:31.683907 20302 hybrid_clock.cc:648] HybridClock initialized: now 1790512771683890 us; error 37 us; skew 500 ppm
I20260927 12:39:31.692456 20302 webserver.cc:460] Webserver started at http://127.0.98.2:33665/ using document root <none> and password file <none>
I20260927 12:39:31.693117 20302 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:31.693192 20302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:31.693403 20302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:31.695731 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data/instance:
uuid: "3783a222449242c79092935f5435eaf6"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "dcdd3a7ddf4ec6e26e36bd4b5cd3d9a8"
server_key_iv: "d72824771f7d16753e8b3e43b5e712fb"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.696377 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3783a222449242c79092935f5435eaf6"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "dcdd3a7ddf4ec6e26e36bd4b5cd3d9a8"
server_key_iv: "d72824771f7d16753e8b3e43b5e712fb"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.700388 20302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:39:31.703071 20316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.704944 20302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:31.705080 20302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal
uuid: "3783a222449242c79092935f5435eaf6"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "dcdd3a7ddf4ec6e26e36bd4b5cd3d9a8"
server_key_iv: "d72824771f7d16753e8b3e43b5e712fb"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.705221 20302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:31.717559 20302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:31.718314 20302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:31.718528 20302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:31.719169 20302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:31.720255 20302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:31.720306 20302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.720351 20302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:31.720367 20302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.729619 20302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36997
I20260927 12:39:31.729758 20429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36997 every 8 connection(s)
I20260927 12:39:31.730960 20302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:39:31.735368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20302
I20260927 12:39:31.735581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:39:31.735916 392 external_mini_cluster.cc:1428] Setting key f6f71057f564ecc8441c976176f9f382
I20260927 12:39:31.738785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--builtin_ntp_servers=127.0.98.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:39:31.741865 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.733268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36731 (local address 127.0.98.62:37431)
0927 12:39:31.733400 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.733404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.734865 (+ 1461us) server_negotiation.cc:407] Connection header received
0927 12:39:31.736676 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.736681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.736750 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.736824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:31.737852 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.738369 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.739774 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.740024 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.740162 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.741356 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:31.741360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:31.741362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:31.741377 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:31.741399 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:31.741409 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:31.741470 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.741622 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.741704 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:39:31.742892 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.733722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:36731)
0927 12:39:31.734619 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.734668 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:31.736403 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:31.737053 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:31.737064 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:31.737165 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:31.737700 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.737714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.738511 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.738514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.739619 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.739635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.740213 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.740217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.740416 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.741049 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:31.741051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:31.741099 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:31.741181 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:31.741483 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:31.741493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:31.741499 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:39:31.741615 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 12:39:31.741960 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":219,"threads_started":1}
I20260927 12:39:31.744860 20430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 12:39:31.745193 20430 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:31.745838 20430 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 12:39:31.747226 20110 ts_manager.cc:194] Registered new tserver with Master: 3783a222449242c79092935f5435eaf6 (127.0.98.2:36997)
I20260927 12:39:31.748055 20110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36731
W20260927 12:39:31.864010 20433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:31.864358 20433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:31.864400 20433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:31.868673 20433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:31.868743 20433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:31.868801 20433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:39:31.873831 20433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:31.874986 20433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:31.876541 20433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:31.886557 20439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.886576 20438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:31.886574 20441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:31.886853 20433 server_base.cc:1034] running on GCE node
I20260927 12:39:31.887454 20433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:31.888170 20433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:31.889405 20433 hybrid_clock.cc:648] HybridClock initialized: now 1790512771889385 us; error 37 us; skew 500 ppm
I20260927 12:39:31.891858 20433 webserver.cc:460] Webserver started at http://127.0.98.3:35001/ using document root <none> and password file <none>
I20260927 12:39:31.892516 20433 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:31.892583 20433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:31.892781 20433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:31.895068 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data/instance:
uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "9059990b6028b86b1dccb548c025ef03"
server_key_iv: "b69f7aed9a720058cff0d5237ab87c73"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.895659 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal/instance:
uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "9059990b6028b86b1dccb548c025ef03"
server_key_iv: "b69f7aed9a720058cff0d5237ab87c73"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.899598 20433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:39:31.902542 20447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.903959 20433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:31.904081 20433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal
uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9"
format_stamp: "Formatted at 2026-09-27 12:39:31 on dist-test-slave-tzgx"
server_key: "9059990b6028b86b1dccb548c025ef03"
server_key_iv: "b69f7aed9a720058cff0d5237ab87c73"
server_key_version: "encryptionkey@0"
I20260927 12:39:31.904259 20433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:31.918081 20433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:31.918807 20433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:31.919001 20433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:31.919624 20433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:39:31.920676 20433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:39:31.920737 20433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.920789 20433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:39:31.920820 20433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:31.931237 20433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39491
I20260927 12:39:31.931281 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39491 every 8 connection(s)
I20260927 12:39:31.932269 20433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:39:31.938903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20433
I20260927 12:39:31.939025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:39:31.939275 392 external_mini_cluster.cc:1428] Setting key ba73b3214a02924137e69f62ea0fc529
I20260927 12:39:31.942911 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.934465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39013 (local address 127.0.98.62:37431)
0927 12:39:31.934652 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.934657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.935949 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 12:39:31.937449 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.937452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.937504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.937589 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:31.938849 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.939628 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.940979 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.941196 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.941284 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.942434 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:31.942438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:31.942440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:31.942453 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:31.942472 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:31.942480 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:31.942540 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.942688 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.942760 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 12:39:31.943819 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.934883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:39013)
0927 12:39:31.935777 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.935827 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:31.937184 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:31.937809 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:31.937817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:31.937912 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:31.938604 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.938631 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.939787 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.939790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.940835 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.940848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.941355 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.941361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.941528 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.942137 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:39:31.942139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:31.942186 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:39:31.942269 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:31.942554 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:31.942564 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:31.942570 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:39:31.942682 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 12:39:31.943020 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":140,"threads_started":1}
I20260927 12:39:31.945041 20561 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 12:39:31.945268 20561 heartbeater.cc:464] Registering TS with master...
I20260927 12:39:31.945802 20561 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 12:39:31.946920 20110 ts_manager.cc:194] Registered new tserver with Master: 748d4f77e2b04b67b0b86d9c4d2c10a9 (127.0.98.3:39491)
I20260927 12:39:31.947485 20110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39013
I20260927 12:39:31.953764 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:39:31.959954 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.955997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.0.98.62:37431)
0927 12:39:31.956179 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.956184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.956236 (+ 52us) server_negotiation.cc:407] Connection header received
0927 12:39:31.956419 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.956422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.956471 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.956558 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:31.956918 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.957507 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.958474 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.958697 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.958786 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.959585 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:31.959589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:31.959591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:31.959604 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:31.959624 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:31.959633 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:31.959689 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.959751 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.959805 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:39:31.962744 20110 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:39:31.965917 20110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:39:31.980870 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.974950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43875 (local address 127.0.0.1:34460)
0927 12:39:31.976039 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.976121 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:31.976330 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:31.976991 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:31.976996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:31.977014 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:31.977126 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.977130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.978205 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.978208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.979279 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.979287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.980260 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.980263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.980377 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.980433 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 12:39:31.980546 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 12:39:31.980694 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"spinlock_wait_cycles":3328,"thread_start_us":114,"threads_started":1}
I20260927 12:39:31.980891 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.975098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.0.98.1:43875)
0927 12:39:31.975835 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.975847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.976207 (+ 360us) server_negotiation.cc:407] Connection header received
0927 12:39:31.976520 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.976526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.976723 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.976828 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:31.977242 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.978084 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.979435 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.980058 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.980173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.980408 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.980597 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.980716 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":126,"threads_started":1}
I20260927 12:39:31.981840 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.975263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.0.98.3:39491)
0927 12:39:31.975765 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.975778 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.976214 (+ 436us) server_negotiation.cc:407] Connection header received
0927 12:39:31.976519 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.976525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.976729 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.976893 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:31.977309 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.978597 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.980199 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.980984 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.981096 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.981276 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.981564 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.981670 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":192,"threads_started":1}
I20260927 12:39:31.981958 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.976165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.98.2:36997)
0927 12:39:31.976872 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:31.976883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:31.976904 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:39:31.976972 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:31.976976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:31.977137 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:31.977270 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:31.977553 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.978641 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.980462 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.981239 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.981337 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.981531 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:31.981718 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 12:39:31.981813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":139,"threads_started":1}
I20260927 12:39:31.982223 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.975113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39491 (local address 127.0.0.1:42264)
0927 12:39:31.976039 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.976122 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:31.976330 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:31.977071 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:31.977074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:31.977090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:31.977193 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.977199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.978752 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.978755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.980035 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.980046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.981308 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.981312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.981401 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.981424 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:39:31.982012 (+ 588us) client_negotiation.cc:240] Negotiation successful
0927 12:39:31.982096 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":99,"spinlock_wait_cycles":6272,"thread_start_us":130,"threads_started":1}
I20260927 12:39:31.981875 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:31.975743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36997 (local address 127.0.0.1:38786)
0927 12:39:31.976550 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:31.976562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:31.976637 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:31.977241 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:31.977244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:31.977260 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:31.977341 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.977346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.978754 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.978756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.980315 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:31.980323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:31.981479 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:31.981483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:31.981572 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:31.981595 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:39:31.981671 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:39:31.981747 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":333,"thread_start_us":83,"threads_started":1}
I20260927 12:39:31.984912 20364 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5307d5bad74585949b6ff0e0891961 (DEFAULT_TABLE table=test-table [id=d66fa1758c60493bbb82855d8843e48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:31.984766 20233 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5307d5bad74585949b6ff0e0891961 (DEFAULT_TABLE table=test-table [id=d66fa1758c60493bbb82855d8843e48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:31.984969 20495 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5307d5bad74585949b6ff0e0891961 (DEFAULT_TABLE table=test-table [id=d66fa1758c60493bbb82855d8843e48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:39:31.985898 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5307d5bad74585949b6ff0e0891961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:31.985903 20495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5307d5bad74585949b6ff0e0891961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:31.985904 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5307d5bad74585949b6ff0e0891961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:31.993232 20578 tablet_bootstrap.cc:492] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Bootstrap starting.
I20260927 12:39:31.994554 20580 tablet_bootstrap.cc:492] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: Bootstrap starting.
I20260927 12:39:31.994696 20579 tablet_bootstrap.cc:492] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Bootstrap starting.
I20260927 12:39:31.996016 20578 tablet_bootstrap.cc:654] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:31.996879 20578 log.cc:824] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:31.996969 20579 tablet_bootstrap.cc:654] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:31.996969 20580 tablet_bootstrap.cc:654] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:31.997838 20579 log.cc:824] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:31.997843 20580 log.cc:824] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:31.999136 20578 tablet_bootstrap.cc:492] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: No bootstrap required, opened a new log
I20260927 12:39:31.999346 20578 ts_tablet_manager.cc:1397] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:39:31.999848 20579 tablet_bootstrap.cc:492] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: No bootstrap required, opened a new log
I20260927 12:39:31.999859 20580 tablet_bootstrap.cc:492] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: No bootstrap required, opened a new log
I20260927 12:39:32.000065 20580 ts_tablet_manager.cc:1397] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:39:32.000065 20579 ts_tablet_manager.cc:1397] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:39:32.003098 20578 raft_consensus.cc:348] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.003546 20578 raft_consensus.cc:374] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:32.003633 20578 raft_consensus.cc:729] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783a222449242c79092935f5435eaf6, State: Initialized, Role: FOLLOWER
I20260927 12:39:32.004164 20578 consensus_queue.cc:260] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.004771 20579 raft_consensus.cc:348] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.004776 20580 raft_consensus.cc:348] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.005039 20430 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 12:39:32.005066 20580 raft_consensus.cc:374] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:32.005066 20579 raft_consensus.cc:374] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:32.005117 20579 raft_consensus.cc:729] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748d4f77e2b04b67b0b86d9c4d2c10a9, State: Initialized, Role: FOLLOWER
I20260927 12:39:32.005117 20580 raft_consensus.cc:729] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5eb99b68eb4e3a894878d655fefaf9, State: Initialized, Role: FOLLOWER
I20260927 12:39:32.005431 20578 ts_tablet_manager.cc:1428] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:39:32.005647 20579 consensus_queue.cc:260] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.005646 20580 consensus_queue.cc:260] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.006718 20579 ts_tablet_manager.cc:1428] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:32.006997 20561 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 12:39:32.007368 20580 ts_tablet_manager.cc:1428] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:39:32.007431 20299 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
W20260927 12:39:32.034274 20300 tablet.cc:2369] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:39:32.063818 20584 raft_consensus.cc:482] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:39:32.064042 20584 raft_consensus.cc:504] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.065228 20584 leader_election.cc:288] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f5eb99b68eb4e3a894878d655fefaf9 (127.0.98.1:43875), 748d4f77e2b04b67b0b86d9c4d2c10a9 (127.0.98.3:39491)
I20260927 12:39:32.068846 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.065782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40653 (local address 127.0.98.3:39491)
0927 12:39:32.065910 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:32.065913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:32.065925 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:39:32.065973 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:32.065975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:32.066014 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:32.066078 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:32.066358 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.066854 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.067891 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.068330 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.068425 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:32.068489 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:32.068653 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 12:39:32.068726 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:39:32.068846 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.065530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39491 (local address 127.0.98.2:40653)
0927 12:39:32.065652 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:32.065668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:32.065761 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:32.066108 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:32.066111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:32.066130 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:32.066243 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:32.066249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.066963 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.066966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:32.067751 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:32.067760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.068443 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.068446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:32.068542 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:32.068560 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:39:32.068623 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:39:32.068696 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"spinlock_wait_cycles":896}
I20260927 12:39:32.069157 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.065708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49169 (local address 127.0.98.1:43875)
0927 12:39:32.065842 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:32.065847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:32.066133 (+ 286us) server_negotiation.cc:407] Connection header received
0927 12:39:32.066300 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:32.066303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:32.066352 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:32.066420 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:39:32.066692 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.067182 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.068199 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.068608 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.068702 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:32.068771 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:32.068945 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 12:39:32.069031 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:39:32.069331 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.065530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43875 (local address 127.0.98.2:49169)
0927 12:39:32.066027 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:32.066039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:32.066157 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:32.066444 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:32.066446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:32.066462 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:39:32.066586 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:32.066592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.067298 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.067302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:32.068059 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:32.068067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.068733 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.068737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:32.068829 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:32.068851 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:39:32.069043 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 12:39:32.069114 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":84,"thread_start_us":157,"threads_started":1}
I20260927 12:39:32.069566 20515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5307d5bad74585949b6ff0e0891961" candidate_uuid: "3783a222449242c79092935f5435eaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" is_pre_election: true
I20260927 12:39:32.069880 20515 raft_consensus.cc:2454] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3783a222449242c79092935f5435eaf6 in term 0.
I20260927 12:39:32.069880 20253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5307d5bad74585949b6ff0e0891961" candidate_uuid: "3783a222449242c79092935f5435eaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" is_pre_election: true
I20260927 12:39:32.070120 20253 raft_consensus.cc:2454] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3783a222449242c79092935f5435eaf6 in term 0.
I20260927 12:39:32.070361 20319 leader_election.cc:302] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3783a222449242c79092935f5435eaf6, 748d4f77e2b04b67b0b86d9c4d2c10a9; no voters:
I20260927 12:39:32.070624 20584 raft_consensus.cc:2781] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:39:32.070701 20584 raft_consensus.cc:482] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:39:32.070741 20584 raft_consensus.cc:3037] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:32.071774 20584 raft_consensus.cc:504] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.072181 20584 leader_election.cc:288] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [CANDIDATE]: Term 1 election: Requested vote from peers 8f5eb99b68eb4e3a894878d655fefaf9 (127.0.98.1:43875), 748d4f77e2b04b67b0b86d9c4d2c10a9 (127.0.98.3:39491)
I20260927 12:39:32.072542 20253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5307d5bad74585949b6ff0e0891961" candidate_uuid: "3783a222449242c79092935f5435eaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5eb99b68eb4e3a894878d655fefaf9"
I20260927 12:39:32.072557 20515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5307d5bad74585949b6ff0e0891961" candidate_uuid: "3783a222449242c79092935f5435eaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9"
I20260927 12:39:32.072677 20515 raft_consensus.cc:3037] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:32.072659 20253 raft_consensus.cc:3037] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:32.073663 20253 raft_consensus.cc:2454] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3783a222449242c79092935f5435eaf6 in term 1.
I20260927 12:39:32.073745 20515 raft_consensus.cc:2454] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3783a222449242c79092935f5435eaf6 in term 1.
I20260927 12:39:32.074035 20318 leader_election.cc:302] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3783a222449242c79092935f5435eaf6, 8f5eb99b68eb4e3a894878d655fefaf9; no voters:
I20260927 12:39:32.074234 20584 raft_consensus.cc:2781] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:32.074466 20584 raft_consensus.cc:686] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 3783a222449242c79092935f5435eaf6, State: Running, Role: LEADER
I20260927 12:39:32.074837 20584 consensus_queue.cc:237] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } }
I20260927 12:39:32.078878 20111 catalog_manager.cc:5495] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3783a222449242c79092935f5435eaf6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3783a222449242c79092935f5435eaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3783a222449242c79092935f5435eaf6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36997 } health_report { overall_health: HEALTHY } } }
I20260927 12:39:32.107647 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.103973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.2:36997)
0927 12:39:32.104129 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:32.104134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:32.104151 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:39:32.104237 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:32.104240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:32.104305 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:32.104394 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:39:32.104700 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.105240 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.106129 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:32.106367 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:32.106452 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:32.106698 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:39:32.107164 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:39:32.107276 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:32.107450 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 12:39:32.107510 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:39:32.113938 20515 raft_consensus.cc:1261] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 1 FOLLOWER]: Refusing update from remote peer 3783a222449242c79092935f5435eaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:32.113940 20253 raft_consensus.cc:1261] T 1d5307d5bad74585949b6ff0e0891961 P 8f5eb99b68eb4e3a894878d655fefaf9 [term 1 FOLLOWER]: Refusing update from remote peer 3783a222449242c79092935f5435eaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:32.114610 20584 consensus_queue.cc:1035] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748d4f77e2b04b67b0b86d9c4d2c10a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:32.114809 20588 consensus_queue.cc:1035] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f5eb99b68eb4e3a894878d655fefaf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:32.124955 20592 mvcc.cc:205] Tried to move back new op lower bound from 7333940314570842112 to 7333940314421374976. Current Snapshot: MvccSnapshot[applied={T|T < 7333940314570842112 or (T in {7333940314570842112})}]
I20260927 12:39:32.127660 20595 mvcc.cc:205] Tried to move back new op lower bound from 7333940314570842112 to 7333940314421374976. Current Snapshot: MvccSnapshot[applied={T|T < 7333940314570842112}]
I20260927 12:39:32.130504 20111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:32.130698 20111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:39:32.133571 20111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5773b3f288454aed8090a06ddfa58623: Sending DeleteTablet for 3 replicas of tablet 1d5307d5bad74585949b6ff0e0891961
I20260927 12:39:32.134601 20495 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5307d5bad74585949b6ff0e0891961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:32 UTC) from {username='slave'} at 127.0.0.1:42264
I20260927 12:39:32.134661 20233 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5307d5bad74585949b6ff0e0891961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:32 UTC) from {username='slave'} at 127.0.0.1:34460
I20260927 12:39:32.135283 20601 tablet_replica.cc:330] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: stopping tablet replica
I20260927 12:39:32.135329 20364 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5307d5bad74585949b6ff0e0891961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:39:32 UTC) from {username='slave'} at 127.0.0.1:38786
I20260927 12:39:32.135716 20601 raft_consensus.cc:2229] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:39:32.135952 20602 tablet_replica.cc:330] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: stopping tablet replica
I20260927 12:39:32.135979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20171
I20260927 12:39:32.137512 20602 raft_consensus.cc:2229] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:39:32.137768 20601 raft_consensus.cc:2258] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:32.138142 20602 raft_consensus.cc:2258] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:39:32.139657 20602 ts_tablet_manager.cc:1905] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:32.139658 20601 ts_tablet_manager.cc:1905] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:39:32.142639 20602 ts_tablet_manager.cc:1918] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:32.142639 20601 ts_tablet_manager.cc:1918] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:39:32.142760 20601 log.cc:1196] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-2/wal/wals/1d5307d5bad74585949b6ff0e0891961
I20260927 12:39:32.142930 20602 log.cc:1196] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Deleting WAL directory at /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512512765972-392-0/minicluster-data/ts-1/wal/wals/1d5307d5bad74585949b6ff0e0891961
I20260927 12:39:32.143085 20601 ts_tablet_manager.cc:1939] T 1d5307d5bad74585949b6ff0e0891961 P 748d4f77e2b04b67b0b86d9c4d2c10a9: Deleting consensus metadata
I20260927 12:39:32.143363 20602 ts_tablet_manager.cc:1939] T 1d5307d5bad74585949b6ff0e0891961 P 3783a222449242c79092935f5435eaf6: Deleting consensus metadata
W20260927 12:39:32.144002 20096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43875 (error 108)
I20260927 12:39:32.144508 20097 catalog_manager.cc:4841] TS 748d4f77e2b04b67b0b86d9c4d2c10a9 (127.0.98.3:39491): tablet 1d5307d5bad74585949b6ff0e0891961 (table test-table [id=d66fa1758c60493bbb82855d8843e48a]) successfully deleted
I20260927 12:39:32.145242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20302
I20260927 12:39:32.145931 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.145271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43875 (local address 127.0.0.1:34466)
0927 12:39:32.145761 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:32.145837 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260927 12:39:32.146178 20096 catalog_manager.cc:4584] TS 8f5eb99b68eb4e3a894878d655fefaf9 (127.0.98.1:43875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d5307d5bad74585949b6ff0e0891961: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43875: connect: Connection refused (error 111)
I20260927 12:39:32.150933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20433
I20260927 12:39:32.152141 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:32.151232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36997 (local address 127.0.0.1:38818)
0927 12:39:32.151949 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:32.152014 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605}
I20260927 12:39:32.156324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20080
2026-09-27T12:39:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1118 ms)
[----------] 6 tests from EncryptionPolicyTest (10529 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:32 dist-test-slave-tzgx krb5kdc[20606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:32 dist-test-slave-tzgx krb5kdc[20606](info): set up 2 sockets
Sep 27 12:39:32 dist-test-slave-tzgx krb5kdc[20606](info): commencing operation
krb5kdc: starting...
W20260927 12:39:33.136049 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:33 dist-test-slave-tzgx krb5kdc[20606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:33.260900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:33.377110 20619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:33.377509 20619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:33.377593 20619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:33.382058 20619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:33.382148 20619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:33.382172 20619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:33.382193 20619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:33.382215 20619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:33.388164 20619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:33.389302 20619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:33.390380 20619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:33.399369 20627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:33.399477 20625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:33.399449 20619 server_base.cc:1034] running on GCE node
W20260927 12:39:33.399612 20624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:33.400153 20619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:33.401325 20619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:33.402531 20619 hybrid_clock.cc:648] HybridClock initialized: now 1790512773402508 us; error 36 us; skew 500 ppm
Sep 27 12:39:33 dist-test-slave-tzgx krb5kdc[20606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:33.405386 20619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:33.406277 20619 webserver.cc:460] Webserver started at http://127.0.0.1:33847/ using document root <none> and password file <none>
I20260927 12:39:33.406802 20619 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:33.406860 20619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:33.407023 20619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:33.409149 20619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "2eaaba18257341f9ac40164198c298b6"
format_stamp: "Formatted at 2026-09-27 12:39:33 on dist-test-slave-tzgx"
server_key: "a66df533bd37fd3330d5c993b73f7d83"
server_key_iv: "cb25bf7165f04cf7325d4823e2d69a32"
server_key_version: "encryptionkey@0"
I20260927 12:39:33.409677 20619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "2eaaba18257341f9ac40164198c298b6"
format_stamp: "Formatted at 2026-09-27 12:39:33 on dist-test-slave-tzgx"
server_key: "a66df533bd37fd3330d5c993b73f7d83"
server_key_iv: "cb25bf7165f04cf7325d4823e2d69a32"
server_key_version: "encryptionkey@0"
I20260927 12:39:33.413327 20619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 12:39:33.415781 20634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:33.417186 20619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:33.417315 20619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "2eaaba18257341f9ac40164198c298b6"
format_stamp: "Formatted at 2026-09-27 12:39:33 on dist-test-slave-tzgx"
server_key: "a66df533bd37fd3330d5c993b73f7d83"
server_key_iv: "cb25bf7165f04cf7325d4823e2d69a32"
server_key_version: "encryptionkey@0"
I20260927 12:39:33.417436 20619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:33.441113 20619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:33.445013 20619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:33.445210 20619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:33.452651 20619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36165
I20260927 12:39:33.452663 20686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36165 every 8 connection(s)
I20260927 12:39:33.453739 20619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:33.456625 20687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:33.457742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20619
I20260927 12:39:33.457866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:33.458181 392 external_mini_cluster.cc:1428] Setting key 8c47df19971dd7191affe3b99d1557a9
I20260927 12:39:33.464246 20687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6: Bootstrap starting.
I20260927 12:39:33.466658 20687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6: Neither blocks nor log segments found. Creating new log.
Sep 27 12:39:33 dist-test-slave-tzgx krb5kdc[20606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:39:33.467455 20687 log.cc:824] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:33.469669 20687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6: No bootstrap required, opened a new log
I20260927 12:39:33.472285 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:33.459479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39065 (local address 127.0.0.1:36165)
0927 12:39:33.459969 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:33.459989 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:33.460036 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:39:33.460990 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:33.461010 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:33.461380 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:33.461701 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:33.462509 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:33.463516 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:33.464403 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:33.464876 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:33.465020 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:33.467501 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:33.467506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:33.467521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:33.467552 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:33.469232 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:33.469739 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:33.469745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:33.469749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:33.469880 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:33.470190 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:33.470194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:33.470195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:33.470584 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:33.470799 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:33.471092 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 12:39:33.471311 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":113,"threads_started":1}
I20260927 12:39:33.474009 20687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eaaba18257341f9ac40164198c298b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260927 12:39:33.474339 20687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:33.474397 20687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eaaba18257341f9ac40164198c298b6, State: Initialized, Role: FOLLOWER
I20260927 12:39:33.474879 20687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eaaba18257341f9ac40164198c298b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260927 12:39:33.475030 20687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:33.475098 20687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:33.475301 20687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:33.475965 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:33.476467 20687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eaaba18257341f9ac40164198c298b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260927 12:39:33.476846 20687 leader_election.cc:302] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eaaba18257341f9ac40164198c298b6; no voters:
I20260927 12:39:33.477192 20687 leader_election.cc:288] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:33.477288 20692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:33.477502 20692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2eaaba18257341f9ac40164198c298b6, State: Running, Role: LEADER
I20260927 12:39:33.477826 20692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eaaba18257341f9ac40164198c298b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } }
I20260927 12:39:33.478668 20687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:33.479070 20692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eaaba18257341f9ac40164198c298b6. Latest consensus state: current_term: 1 leader_uuid: "2eaaba18257341f9ac40164198c298b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eaaba18257341f9ac40164198c298b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } } }
I20260927 12:39:33.479192 20692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:33.479429 20692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eaaba18257341f9ac40164198c298b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eaaba18257341f9ac40164198c298b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36165 } } }
I20260927 12:39:33.479591 20692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:33.479624 20701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:33.483078 20701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:33.488472 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:33.478428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60617 (local address 127.0.0.1:36165)
0927 12:39:33.478555 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:33.478559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:33.478762 (+ 203us) server_negotiation.cc:407] Connection header received
0927 12:39:33.478935 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:33.478938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:33.478992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:33.479055 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:33.479904 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:33.480669 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:33.481857 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:33.482195 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:33.482318 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:33.484485 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:33.484490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:33.484493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:33.484524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:33.486218 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:33.486875 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:33.486879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:33.486881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:33.486974 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:33.487421 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:33.487424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:33.487427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:33.487699 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:33.487872 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:33.488211 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 12:39:33.488316 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:39:33.489991 20701 catalog_manager.cc:1347] Generated new cluster ID: 9747222fc06f4e349e8ba6b3924e65da
I20260927 12:39:33.490059 20701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:33.508713 20701 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:33.509760 20701 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:33.515419 20701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eaaba18257341f9ac40164198c298b6: Generated new TSK 0
I20260927 12:39:33.516243 20701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:33.524714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20619
W20260927 12:39:33.529663 20604 connection.cc:535] client connection to 127.0.0.1:36165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36165): Connection reset by peer (error 104)
2026-09-27T12:39:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:33 dist-test-slave-tzgx krb5kdc[20718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:33 dist-test-slave-tzgx krb5kdc[20718](info): set up 2 sockets
Sep 27 12:39:33 dist-test-slave-tzgx krb5kdc[20718](info): commencing operation
krb5kdc: starting...
W20260927 12:39:34.496017 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:34 dist-test-slave-tzgx krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:34.624244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:34.744313 20731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:34.744645 20731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:34.744693 20731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:34.748941 20731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:34.749030 20731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:34.749054 20731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:34.749073 20731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:34.749091 20731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:34.754751 20731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:34.755882 20731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:34.756994 20731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:39:34.765826 20731 server_base.cc:1034] running on GCE node
W20260927 12:39:34.765784 20737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:34.765806 20736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:34.765784 20739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:34.766661 20731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:34.767863 20731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:34.769095 20731 hybrid_clock.cc:648] HybridClock initialized: now 1790512774769041 us; error 41 us; skew 500 ppm
Sep 27 12:39:34 dist-test-slave-tzgx krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:34.772262 20731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:34.773262 20731 webserver.cc:460] Webserver started at http://127.0.0.1:43413/ using document root <none> and password file <none>
I20260927 12:39:34.773847 20731 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:34.773917 20731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:34.774133 20731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:34.776351 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "39fa7216a11c4593a8bd02ebdf38b2b6"
format_stamp: "Formatted at 2026-09-27 12:39:34 on dist-test-slave-tzgx"
server_key: "6dba2e92b4f9988bda34ffb988235cab"
server_key_iv: "95bea16c35cca46365d2fba5756b8de7"
server_key_version: "encryptionkey@0"
I20260927 12:39:34.777092 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "39fa7216a11c4593a8bd02ebdf38b2b6"
format_stamp: "Formatted at 2026-09-27 12:39:34 on dist-test-slave-tzgx"
server_key: "6dba2e92b4f9988bda34ffb988235cab"
server_key_iv: "95bea16c35cca46365d2fba5756b8de7"
server_key_version: "encryptionkey@0"
I20260927 12:39:34.781165 20731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:34.783771 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:34.785104 20731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:34.785264 20731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "39fa7216a11c4593a8bd02ebdf38b2b6"
format_stamp: "Formatted at 2026-09-27 12:39:34 on dist-test-slave-tzgx"
server_key: "6dba2e92b4f9988bda34ffb988235cab"
server_key_iv: "95bea16c35cca46365d2fba5756b8de7"
server_key_version: "encryptionkey@0"
I20260927 12:39:34.785382 20731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:34.809813 20731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:34.813436 20731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:34.813634 20731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:34.823587 20731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42393
I20260927 12:39:34.823660 20798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42393 every 8 connection(s)
I20260927 12:39:34.824728 20731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:34.827769 20799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:34.831518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20731
I20260927 12:39:34.831759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:34.832140 392 external_mini_cluster.cc:1428] Setting key 479004b89ed3b2a1f01ed593a2097681
I20260927 12:39:34.835006 20799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6: Bootstrap starting.
I20260927 12:39:34.838181 20799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:34.839175 20799 log.cc:824] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:34.841457 20799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6: No bootstrap required, opened a new log
I20260927 12:39:34.844373 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:34.833774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51129 (local address 127.0.0.1:42393)
0927 12:39:34.834815 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:34.834837 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:34.834890 (+ 53us) server_negotiation.cc:407] Connection header received
0927 12:39:34.835847 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:34.835872 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:34.836416 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:34.836838 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:34.837337 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:34.838947 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:34.839997 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:34.840588 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:34.840752 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:34.841703 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:34.841710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:34.841728 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:34.842304 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:34.842380 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:34.842413 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:34.842563 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:34.842777 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 12:39:34.842921 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":268,"threads_started":1}
I20260927 12:39:34.845762 20799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:39:34.846072 20799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:34.846133 20799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fa7216a11c4593a8bd02ebdf38b2b6, State: Initialized, Role: FOLLOWER
I20260927 12:39:34.846628 20799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:39:34.846807 20799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:34.846903 20799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:34.847039 20799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:34.848202 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:34.848590 20799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:39:34.849036 20799 leader_election.cc:302] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39fa7216a11c4593a8bd02ebdf38b2b6; no voters:
I20260927 12:39:34.849427 20799 leader_election.cc:288] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:34.849551 20804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:34.849857 20804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [term 1 LEADER]: Becoming Leader. State: Replica: 39fa7216a11c4593a8bd02ebdf38b2b6, State: Running, Role: LEADER
I20260927 12:39:34.850265 20804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:39:34.850975 20799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:34.852056 20810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39fa7216a11c4593a8bd02ebdf38b2b6. Latest consensus state: current_term: 1 leader_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260927 12:39:34.852226 20810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:34.852635 20818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:34.852656 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fa7216a11c4593a8bd02ebdf38b2b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260927 12:39:34.852823 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:34.856467 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:34.850546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37045 (local address 127.0.0.1:42393)
0927 12:39:34.850662 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:34.850667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:34.851108 (+ 441us) server_negotiation.cc:407] Connection header received
0927 12:39:34.851311 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:34.851314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:34.851371 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:34.851462 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:34.851816 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:34.852731 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:34.854191 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:34.854577 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:34.854686 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:34.855956 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:34.855960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:34.855963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:34.855978 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:34.856006 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:34.856018 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:34.856115 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:34.856274 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 12:39:34.856314 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 12:39:34.856669 20818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:34.864712 20818 catalog_manager.cc:1347] Generated new cluster ID: fb407bd6221445548d80d714a052f2e8
I20260927 12:39:34.864799 20818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:34.874106 20818 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:34.875519 20818 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:34.883435 20818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39fa7216a11c4593a8bd02ebdf38b2b6: Generated new TSK 0
I20260927 12:39:34.884514 20818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:34.890031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20731
W20260927 12:39:34.894587 20716 connection.cc:535] client connection to 127.0.0.1:42393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42393): Connection reset by peer (error 104)
2026-09-27T12:39:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:34 dist-test-slave-tzgx krb5kdc[20830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:34 dist-test-slave-tzgx krb5kdc[20830](info): set up 2 sockets
Sep 27 12:39:34 dist-test-slave-tzgx krb5kdc[20830](info): commencing operation
krb5kdc: starting...
W20260927 12:39:35.857113 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:35 dist-test-slave-tzgx krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:36.001189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:36.121841 20843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:36.122141 20843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:36.122190 20843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:36.126403 20843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:36.126485 20843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:36.126503 20843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:36.126518 20843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:36.126533 20843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:36.132840 20843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:36.133937 20843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:36.134941 20843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:36.144047 20849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:36.144265 20843 server_base.cc:1034] running on GCE node
W20260927 12:39:36.144109 20851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:36.144083 20848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:36.144999 20843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:36.146209 20843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:36.147413 20843 hybrid_clock.cc:648] HybridClock initialized: now 1790512776147388 us; error 50 us; skew 500 ppm
Sep 27 12:39:36 dist-test-slave-tzgx krb5kdc[20830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:36.150496 20843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:36.151540 20843 webserver.cc:460] Webserver started at http://127.0.0.1:35743/ using document root <none> and password file <none>
I20260927 12:39:36.152149 20843 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:36.152226 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:36.152434 20843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:36.154546 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "e2676a2151604022806acb9c34cbdd64"
format_stamp: "Formatted at 2026-09-27 12:39:36 on dist-test-slave-tzgx"
server_key: "dace69c241d5aecda900e5de992bfb94"
server_key_iv: "84dbc947a12635f9969c05243f44bb97"
server_key_version: "encryptionkey@0"
I20260927 12:39:36.155134 20843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2676a2151604022806acb9c34cbdd64"
format_stamp: "Formatted at 2026-09-27 12:39:36 on dist-test-slave-tzgx"
server_key: "dace69c241d5aecda900e5de992bfb94"
server_key_iv: "84dbc947a12635f9969c05243f44bb97"
server_key_version: "encryptionkey@0"
I20260927 12:39:36.159009 20843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:39:36.161935 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:36.163296 20843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:36.163436 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "e2676a2151604022806acb9c34cbdd64"
format_stamp: "Formatted at 2026-09-27 12:39:36 on dist-test-slave-tzgx"
server_key: "dace69c241d5aecda900e5de992bfb94"
server_key_iv: "84dbc947a12635f9969c05243f44bb97"
server_key_version: "encryptionkey@0"
I20260927 12:39:36.163632 20843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:36.168651 20843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:36.172241 20843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:36.172436 20843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:36.180814 20843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45535
I20260927 12:39:36.180926 20910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45535 every 8 connection(s)
I20260927 12:39:36.181901 20843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:36.185247 20911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:36.188402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20843
I20260927 12:39:36.188536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:36.188799 392 external_mini_cluster.cc:1428] Setting key f0e443e86bff84e7832acff4b301d1be
I20260927 12:39:36.192595 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64: Bootstrap starting.
I20260927 12:39:36.194769 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:36.190271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45285 (local address 127.0.0.1:45535)
0927 12:39:36.190741 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:36.190757 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:36.190796 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:39:36.191520 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:36.191542 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:36.191894 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:36.192299 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:36.193388 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:36.193393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:36.193407 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:36.193418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:36.193479 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:36.193502 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:36.193638 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:36.193772 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 12:39:36.193954 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260927 12:39:36.195818 20911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:36.196637 20911 log.cc:824] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:36.197975 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:36.199208 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64: No bootstrap required, opened a new log
I20260927 12:39:36.202067 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:36.200314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33039 (local address 127.0.0.1:45535)
0927 12:39:36.200448 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:36.200453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:36.200690 (+ 237us) server_negotiation.cc:407] Connection header received
0927 12:39:36.200811 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:36.200813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:36.200847 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:36.200916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:36.201638 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:36.201641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:36.201643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:36.201654 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:36.201682 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:36.201694 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:36.201788 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:36.201888 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 12:39:36.201931 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:39:36.203675 20911 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2676a2151604022806acb9c34cbdd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45535 } }
I20260927 12:39:36.204039 20911 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:36.204192 20911 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2676a2151604022806acb9c34cbdd64, State: Initialized, Role: FOLLOWER
I20260927 12:39:36.204649 20911 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2676a2151604022806acb9c34cbdd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45535 } }
I20260927 12:39:36.204818 20911 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:36.204887 20911 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:36.204985 20911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:36.205952 20911 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2676a2151604022806acb9c34cbdd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45535 } }
I20260927 12:39:36.206305 20911 leader_election.cc:302] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2676a2151604022806acb9c34cbdd64; no voters:
I20260927 12:39:36.206601 20911 leader_election.cc:288] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:36.206700 20921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:36.206908 20921 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [term 1 LEADER]: Becoming Leader. State: Replica: e2676a2151604022806acb9c34cbdd64, State: Running, Role: LEADER
I20260927 12:39:36.207360 20921 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2676a2151604022806acb9c34cbdd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45535 } }
I20260927 12:39:36.207911 20911 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:36.210033 20923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2676a2151604022806acb9c34cbdd64. Latest consensus state: current_term: 1 leader_uuid: "e2676a2151604022806acb9c34cbdd64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2676a2151604022806acb9c34cbdd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45535 } } }
I20260927 12:39:36.210187 20923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:36.210644 20922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2676a2151604022806acb9c34cbdd64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2676a2151604022806acb9c34cbdd64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45535 } } }
I20260927 12:39:36.211094 20922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64 [sys.catalog]: This master's current role is: LEADER
W20260927 12:39:36.213097 20935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:36.213179 20935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:36.213255 20936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:36.214129 20936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:36.220333 20936 catalog_manager.cc:1347] Generated new cluster ID: 5d42cfbf748e4820be371a03c246ca5f
I20260927 12:39:36.220417 20936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:36.234107 20936 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:36.234437 20936 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:36.247107 20936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2676a2151604022806acb9c34cbdd64: Generated new TSK 0
I20260927 12:39:36.248023 20936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:36.259847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20843
2026-09-27T12:39:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:36 dist-test-slave-tzgx krb5kdc[20942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:36 dist-test-slave-tzgx krb5kdc[20942](info): set up 2 sockets
Sep 27 12:39:36 dist-test-slave-tzgx krb5kdc[20942](info): commencing operation
krb5kdc: starting...
W20260927 12:39:37.236146 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:37 dist-test-slave-tzgx krb5kdc[20942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:37.365590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:37.487085 20955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:37.487406 20955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:37.487466 20955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:37.491686 20955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:37.491780 20955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:37.491807 20955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:37.491828 20955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:37.491849 20955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:37.497673 20955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:37.498844 20955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:37.499891 20955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:37.508721 20960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:37.508659 20963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:37.508687 20955 server_base.cc:1034] running on GCE node
W20260927 12:39:37.508700 20961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:37.509624 20955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:37.510861 20955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:37.512130 20955 hybrid_clock.cc:648] HybridClock initialized: now 1790512777512068 us; error 43 us; skew 500 ppm
Sep 27 12:39:37 dist-test-slave-tzgx krb5kdc[20942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:37.522740 20955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:37.523800 20955 webserver.cc:460] Webserver started at http://127.0.0.1:43743/ using document root <none> and password file <none>
I20260927 12:39:37.524447 20955 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:37.524523 20955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:37.524727 20955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:37.526857 20955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "012fcc26022542329a5aff913e482429"
format_stamp: "Formatted at 2026-09-27 12:39:37 on dist-test-slave-tzgx"
server_key: "9f659bf7c17a9548fd229874011e8d28"
server_key_iv: "26491f823cdebae9c05bcd0b35143c1b"
server_key_version: "encryptionkey@0"
I20260927 12:39:37.527432 20955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "012fcc26022542329a5aff913e482429"
format_stamp: "Formatted at 2026-09-27 12:39:37 on dist-test-slave-tzgx"
server_key: "9f659bf7c17a9548fd229874011e8d28"
server_key_iv: "26491f823cdebae9c05bcd0b35143c1b"
server_key_version: "encryptionkey@0"
I20260927 12:39:37.531631 20955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:39:37.534602 20970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:37.536265 20955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:37.536406 20955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "012fcc26022542329a5aff913e482429"
format_stamp: "Formatted at 2026-09-27 12:39:37 on dist-test-slave-tzgx"
server_key: "9f659bf7c17a9548fd229874011e8d28"
server_key_iv: "26491f823cdebae9c05bcd0b35143c1b"
server_key_version: "encryptionkey@0"
I20260927 12:39:37.536550 20955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:37.567175 20955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:37.570566 20955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:37.570746 20955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:37.578159 20955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35413
I20260927 12:39:37.578186 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35413 every 8 connection(s)
I20260927 12:39:37.579172 20955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:37.582317 21023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:37.585486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 20955
I20260927 12:39:37.585618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:37.585955 392 external_mini_cluster.cc:1428] Setting key b54fb1ddeb50bf62d708b25e2b34a702
I20260927 12:39:37.589011 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429: Bootstrap starting.
I20260927 12:39:37.591881 21023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:37.592821 21023 log.cc:824] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:37.595453 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429: No bootstrap required, opened a new log
Sep 27 12:39:37 dist-test-slave-tzgx krb5kdc[20942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:39:37.599483 21023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012fcc26022542329a5aff913e482429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35413 } }
I20260927 12:39:37.599849 21023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:37.599905 21023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012fcc26022542329a5aff913e482429, State: Initialized, Role: FOLLOWER
I20260927 12:39:37.600409 21023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012fcc26022542329a5aff913e482429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35413 } }
I20260927 12:39:37.600584 21023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:37.600654 21023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:37.600759 21023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:37.602027 21023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012fcc26022542329a5aff913e482429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35413 } }
I20260927 12:39:37.602416 21023 leader_election.cc:302] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012fcc26022542329a5aff913e482429; no voters:
I20260927 12:39:37.602774 21023 leader_election.cc:288] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:37.602885 21028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:37.603142 21028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [term 1 LEADER]: Becoming Leader. State: Replica: 012fcc26022542329a5aff913e482429, State: Running, Role: LEADER
I20260927 12:39:37.603488 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:37.587573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45485 (local address 127.0.0.1:35413)
0927 12:39:37.588073 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:37.588126 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:37.588175 (+ 49us) server_negotiation.cc:407] Connection header received
0927 12:39:37.589058 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:37.589084 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:37.589564 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:37.589956 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:37.590842 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:37.592353 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:37.593502 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:37.594179 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:37.594440 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:37.596965 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:37.596972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:37.596993 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:37.597037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:37.599735 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:37.600481 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:37.600488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:37.600495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:37.600658 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:37.600992 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:37.600996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:37.600998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:37.601556 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:37.601764 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:37.602061 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 12:39:37.602391 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":147,"threads_started":1}
I20260927 12:39:37.603516 21028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012fcc26022542329a5aff913e482429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35413 } }
I20260927 12:39:37.604120 21023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:37.605361 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012fcc26022542329a5aff913e482429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012fcc26022542329a5aff913e482429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35413 } } }
I20260927 12:39:37.605508 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:37.606168 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012fcc26022542329a5aff913e482429. Latest consensus state: current_term: 1 leader_uuid: "012fcc26022542329a5aff913e482429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012fcc26022542329a5aff913e482429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35413 } } }
I20260927 12:39:37.606329 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:37.607045 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 12:39:37.610401 21043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:37.610553 21043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:37.610731 21037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:37.611646 21037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:37.617282 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:37.609320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39341 (local address 127.0.0.1:35413)
0927 12:39:37.609429 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:37.609434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:37.609695 (+ 261us) server_negotiation.cc:407] Connection header received
0927 12:39:37.609828 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:37.609832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:37.609885 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:37.609960 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:37.610783 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:37.611304 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:37.612114 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:37.612390 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:37.612482 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:37.613808 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:37.613813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:37.613816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:37.613846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:37.615518 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:37.615961 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:37.615965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:37.615966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:37.616059 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:37.616432 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:37.616436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:37.616437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:37.616647 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:37.616764 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:37.616992 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 12:39:37.617110 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:39:37.617923 21037 catalog_manager.cc:1347] Generated new cluster ID: bd07504dcd9d41498e4b2bbaf3133b73
I20260927 12:39:37.617977 21037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:37.629536 21037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:37.631006 21037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:37.637843 21037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 012fcc26022542329a5aff913e482429: Generated new TSK 0
I20260927 12:39:37.638780 21037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:37.642606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 20955
2026-09-27T12:39:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:37 dist-test-slave-tzgx krb5kdc[21054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:37 dist-test-slave-tzgx krb5kdc[21054](info): set up 2 sockets
Sep 27 12:39:37 dist-test-slave-tzgx krb5kdc[21054](info): commencing operation
krb5kdc: starting...
W20260927 12:39:38.608058 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:38 dist-test-slave-tzgx krb5kdc[21054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:38.731998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:38.846601 21067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:38.846921 21067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:38.846984 21067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:38.851806 21067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:38.851920 21067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:38.851953 21067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:38.851974 21067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:38.851994 21067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:38.857980 21067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:38.859141 21067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:38.860232 21067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:38.869279 21073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:38.869279 21072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:38.869349 21067 server_base.cc:1034] running on GCE node
W20260927 12:39:38.869279 21075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:38.870172 21067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:38.871335 21067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:38.872488 21067 hybrid_clock.cc:648] HybridClock initialized: now 1790512778872470 us; error 37 us; skew 500 ppm
Sep 27 12:39:38 dist-test-slave-tzgx krb5kdc[21054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:38.875355 21067 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:38.876374 21067 webserver.cc:460] Webserver started at http://127.0.0.1:42459/ using document root <none> and password file <none>
I20260927 12:39:38.876927 21067 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:38.877041 21067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:38.877259 21067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:38.879395 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "1e1edbe9c6ae49edad2994a7d5d98908"
format_stamp: "Formatted at 2026-09-27 12:39:38 on dist-test-slave-tzgx"
server_key: "447026636f86e852c4195746b1368c9d"
server_key_iv: "b88c612d62ac7426caea8a1d501b6af3"
server_key_version: "encryptionkey@0"
I20260927 12:39:38.879976 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e1edbe9c6ae49edad2994a7d5d98908"
format_stamp: "Formatted at 2026-09-27 12:39:38 on dist-test-slave-tzgx"
server_key: "447026636f86e852c4195746b1368c9d"
server_key_iv: "b88c612d62ac7426caea8a1d501b6af3"
server_key_version: "encryptionkey@0"
I20260927 12:39:38.884018 21067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 12:39:38.886837 21082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:38.888334 21067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 12:39:38.888481 21067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "1e1edbe9c6ae49edad2994a7d5d98908"
format_stamp: "Formatted at 2026-09-27 12:39:38 on dist-test-slave-tzgx"
server_key: "447026636f86e852c4195746b1368c9d"
server_key_iv: "b88c612d62ac7426caea8a1d501b6af3"
server_key_version: "encryptionkey@0"
I20260927 12:39:38.888615 21067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:38.914390 21067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:38.918111 21067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:38.918321 21067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:38.925907 21067 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40427
I20260927 12:39:38.925935 21134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40427 every 8 connection(s)
I20260927 12:39:38.926926 21067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:38.928989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21067
I20260927 12:39:38.929114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:38.929366 392 external_mini_cluster.cc:1428] Setting key 6e5a0c4945acc278ee337d6c9b1ca6b7
I20260927 12:39:38.930003 21135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:38.936844 21135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908: Bootstrap starting.
I20260927 12:39:38.938735 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:38.930688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42053 (local address 127.0.0.1:40427)
0927 12:39:38.931174 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:38.931190 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:38.931229 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:39:38.931884 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:38.931904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:38.932321 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:38.932670 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:38.933072 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:38.934108 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:38.935290 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:38.935797 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:38.935926 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:38.936841 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:38.936847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:38.936861 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:38.937317 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:38.937380 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:38.937405 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:38.937488 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:38.937639 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 12:39:38.937854 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260927 12:39:38.939594 21135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:38.940382 21135 log.cc:824] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:38.941888 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:38.942731 21135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908: No bootstrap required, opened a new log
I20260927 12:39:38.947096 21135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260927 12:39:38.947391 21135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:38.947441 21135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1edbe9c6ae49edad2994a7d5d98908, State: Initialized, Role: FOLLOWER
I20260927 12:39:38.947923 21135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260927 12:39:38.948140 21135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:38.948169 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:38.944020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41345 (local address 127.0.0.1:40427)
0927 12:39:38.944158 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:38.944162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:38.944334 (+ 172us) server_negotiation.cc:407] Connection header received
0927 12:39:38.944474 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:38.944478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:38.944524 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:38.944592 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:38.944858 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:38.945400 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:38.946461 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:38.946729 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:38.946829 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:38.947610 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:38.947614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:38.947616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:38.947629 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:38.947660 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:38.947672 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:38.947742 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:38.947868 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 12:39:38.947922 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:39:38.948220 21135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:38.948323 21135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:38.949440 21135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260927 12:39:38.949772 21135 leader_election.cc:302] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e1edbe9c6ae49edad2994a7d5d98908; no voters:
I20260927 12:39:38.950078 21135 leader_election.cc:288] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:38.950275 21145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:38.950529 21145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [term 1 LEADER]: Becoming Leader. State: Replica: 1e1edbe9c6ae49edad2994a7d5d98908, State: Running, Role: LEADER
I20260927 12:39:38.950831 21145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } }
I20260927 12:39:38.951297 21135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:38.952440 21146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } } }
I20260927 12:39:38.952601 21146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:38.952986 21147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e1edbe9c6ae49edad2994a7d5d98908. Latest consensus state: current_term: 1 leader_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1edbe9c6ae49edad2994a7d5d98908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40427 } } }
I20260927 12:39:38.953138 21147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908 [sys.catalog]: This master's current role is: LEADER
W20260927 12:39:38.956197 21160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:38.956279 21160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:38.956348 21154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:38.957084 21154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:38.962852 21154 catalog_manager.cc:1347] Generated new cluster ID: ab82ebbea67d4da69724ab02c9c407c5
I20260927 12:39:38.962944 21154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:38.993752 21154 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:38.995110 21154 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:39.007236 21154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e1edbe9c6ae49edad2994a7d5d98908: Generated new TSK 0
I20260927 12:39:39.008387 21154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:39.020612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21067
2026-09-27T12:39:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:39 dist-test-slave-tzgx krb5kdc[21166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:39 dist-test-slave-tzgx krb5kdc[21166](info): set up 2 sockets
Sep 27 12:39:39 dist-test-slave-tzgx krb5kdc[21166](info): commencing operation
krb5kdc: starting...
W20260927 12:39:40.004179 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:40 dist-test-slave-tzgx krb5kdc[21166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:40.133472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:40.254742 21179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:40.255061 21179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:40.255103 21179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:40.259259 21179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:40.259347 21179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:40.259367 21179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:40.259382 21179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:40.259397 21179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:40.265508 21179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:40.266662 21179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:40.267654 21179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:40.276608 21184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:40.276809 21187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:40.276713 21185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:40.276898 21179 server_base.cc:1034] running on GCE node
I20260927 12:39:40.277660 21179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:40.278945 21179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:40.280211 21179 hybrid_clock.cc:648] HybridClock initialized: now 1790512780280181 us; error 45 us; skew 500 ppm
Sep 27 12:39:40 dist-test-slave-tzgx krb5kdc[21166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:40.283238 21179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:40.284267 21179 webserver.cc:460] Webserver started at http://127.0.0.1:34397/ using document root <none> and password file <none>
I20260927 12:39:40.284834 21179 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:40.284900 21179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:40.285104 21179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:40.287297 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "6950233f0a8942498d6cd0f57065fd1d"
format_stamp: "Formatted at 2026-09-27 12:39:40 on dist-test-slave-tzgx"
server_key: "9823bb67d3f697855e455f8a2ef451ca"
server_key_iv: "73138d3a683fd482a3a4e240af152d6f"
server_key_version: "encryptionkey@0"
I20260927 12:39:40.287910 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "6950233f0a8942498d6cd0f57065fd1d"
format_stamp: "Formatted at 2026-09-27 12:39:40 on dist-test-slave-tzgx"
server_key: "9823bb67d3f697855e455f8a2ef451ca"
server_key_iv: "73138d3a683fd482a3a4e240af152d6f"
server_key_version: "encryptionkey@0"
I20260927 12:39:40.292363 21179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:39:40.295250 21194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:40.296845 21179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:40.296994 21179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "6950233f0a8942498d6cd0f57065fd1d"
format_stamp: "Formatted at 2026-09-27 12:39:40 on dist-test-slave-tzgx"
server_key: "9823bb67d3f697855e455f8a2ef451ca"
server_key_iv: "73138d3a683fd482a3a4e240af152d6f"
server_key_version: "encryptionkey@0"
I20260927 12:39:40.297154 21179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:40.302217 21179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:40.306064 21179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:40.306291 21179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:40.313946 21179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43937
I20260927 12:39:40.313992 21246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43937 every 8 connection(s)
I20260927 12:39:40.315045 21179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:40.318195 21247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:40.321051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21179
I20260927 12:39:40.321218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:40.321557 392 external_mini_cluster.cc:1428] Setting key b209914df9dcbdaf746f75a004de7be0
I20260927 12:39:40.325683 21247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d: Bootstrap starting.
I20260927 12:39:40.328455 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:40.323544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50053 (local address 127.0.0.1:43937)
0927 12:39:40.324364 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:40.324383 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:40.324426 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:39:40.325128 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:40.325148 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:40.325486 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:40.325829 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:40.326818 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:40.326824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:40.326841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:40.326854 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:40.326926 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:40.326949 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:40.327165 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:40.327392 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:39:40.327603 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":353,"threads_started":1}
I20260927 12:39:40.328874 21247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:40.329689 21247 log.cc:824] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:40.331811 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:40.332283 21247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d: No bootstrap required, opened a new log
I20260927 12:39:40.336020 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:40.334293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51891 (local address 127.0.0.1:43937)
0927 12:39:40.334446 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:40.334451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:40.334542 (+ 91us) server_negotiation.cc:407] Connection header received
0927 12:39:40.334652 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:40.334655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:40.334697 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:40.334801 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:40.335513 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:40.335517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:40.335521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:40.335534 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:40.335568 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:40.335582 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:40.335695 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:40.335820 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 12:39:40.335867 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:39:40.337182 21247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6950233f0a8942498d6cd0f57065fd1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 12:39:40.337453 21247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:40.337488 21247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6950233f0a8942498d6cd0f57065fd1d, State: Initialized, Role: FOLLOWER
I20260927 12:39:40.337955 21247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6950233f0a8942498d6cd0f57065fd1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 12:39:40.338089 21247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:40.338186 21247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:40.338300 21247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:40.339314 21247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6950233f0a8942498d6cd0f57065fd1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 12:39:40.339674 21247 leader_election.cc:302] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6950233f0a8942498d6cd0f57065fd1d; no voters:
I20260927 12:39:40.339994 21247 leader_election.cc:288] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:40.340190 21257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:40.340418 21257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [term 1 LEADER]: Becoming Leader. State: Replica: 6950233f0a8942498d6cd0f57065fd1d, State: Running, Role: LEADER
I20260927 12:39:40.340749 21257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6950233f0a8942498d6cd0f57065fd1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 12:39:40.341321 21247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:40.342857 21258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6950233f0a8942498d6cd0f57065fd1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6950233f0a8942498d6cd0f57065fd1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } } }
I20260927 12:39:40.342901 21259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6950233f0a8942498d6cd0f57065fd1d. Latest consensus state: current_term: 1 leader_uuid: "6950233f0a8942498d6cd0f57065fd1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6950233f0a8942498d6cd0f57065fd1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } } }
I20260927 12:39:40.343101 21259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:40.343099 21258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:40.343510 21271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:40.346947 21271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:40.353446 21271 catalog_manager.cc:1347] Generated new cluster ID: 51bd87b4644d4137bb8423ee0b400ba9
I20260927 12:39:40.353549 21271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:40.363618 21271 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:40.364037 21271 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:40.370208 21271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6950233f0a8942498d6cd0f57065fd1d: Generated new TSK 0
I20260927 12:39:40.371193 21271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:40.382763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21179
2026-09-27T12:39:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:40 dist-test-slave-tzgx krb5kdc[21278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:40 dist-test-slave-tzgx krb5kdc[21278](info): set up 2 sockets
Sep 27 12:39:40 dist-test-slave-tzgx krb5kdc[21278](info): commencing operation
krb5kdc: starting...
W20260927 12:39:41.368209 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:41 dist-test-slave-tzgx krb5kdc[21278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:41.500077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:41.624992 21291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:41.625288 21291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:41.625331 21291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:41.629916 21291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:41.630018 21291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:41.630038 21291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:41.630059 21291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:41.630075 21291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:41.636826 21291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:41.638069 21291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:41.639176 21291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:41.648898 21296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:41.649044 21297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:41.649385 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:41.649444 21291 server_base.cc:1034] running on GCE node
I20260927 12:39:41.650096 21291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:41.651345 21291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:41.652557 21291 hybrid_clock.cc:648] HybridClock initialized: now 1790512781652511 us; error 61 us; skew 500 ppm
Sep 27 12:39:41 dist-test-slave-tzgx krb5kdc[21278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:41.655741 21291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:41.656982 21291 webserver.cc:460] Webserver started at http://127.0.0.1:37569/ using document root <none> and password file <none>
I20260927 12:39:41.657735 21291 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:41.657821 21291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:41.658047 21291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:41.660585 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "6c4176a63c444aa4bde90e4cd25b3e9d"
format_stamp: "Formatted at 2026-09-27 12:39:41 on dist-test-slave-tzgx"
server_key: "a6239a0d36842c640f9da628dad9ee92"
server_key_iv: "fcb348847041e95de1bb2ad781902a06"
server_key_version: "encryptionkey@0"
I20260927 12:39:41.661203 21291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c4176a63c444aa4bde90e4cd25b3e9d"
format_stamp: "Formatted at 2026-09-27 12:39:41 on dist-test-slave-tzgx"
server_key: "a6239a0d36842c640f9da628dad9ee92"
server_key_iv: "fcb348847041e95de1bb2ad781902a06"
server_key_version: "encryptionkey@0"
I20260927 12:39:41.665223 21291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:41.668444 21306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:41.669790 21291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:39:41.669938 21291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "6c4176a63c444aa4bde90e4cd25b3e9d"
format_stamp: "Formatted at 2026-09-27 12:39:41 on dist-test-slave-tzgx"
server_key: "a6239a0d36842c640f9da628dad9ee92"
server_key_iv: "fcb348847041e95de1bb2ad781902a06"
server_key_version: "encryptionkey@0"
I20260927 12:39:41.670064 21291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:41.687839 21291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:41.691138 21291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:41.691337 21291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:41.698402 21291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37011
I20260927 12:39:41.698454 21358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37011 every 8 connection(s)
I20260927 12:39:41.699386 21291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:41.702262 21359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:41.707171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21291
I20260927 12:39:41.707316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:41.707584 392 external_mini_cluster.cc:1428] Setting key 8c09b0271cae064e25b78c02f0f3c4b8
I20260927 12:39:41.707967 21359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d: Bootstrap starting.
I20260927 12:39:41.710470 21359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:41.711373 21359 log.cc:824] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:41.713653 21359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d: No bootstrap required, opened a new log
Sep 27 12:39:41 dist-test-slave-tzgx krb5kdc[21278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:39:41.718050 21359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260927 12:39:41.718366 21359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:41.718447 21359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4176a63c444aa4bde90e4cd25b3e9d, State: Initialized, Role: FOLLOWER
I20260927 12:39:41.719013 21359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260927 12:39:41.719193 21359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:41.719264 21359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:41.719367 21359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:41.720532 21359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260927 12:39:41.720924 21359 leader_election.cc:302] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c4176a63c444aa4bde90e4cd25b3e9d; no voters:
I20260927 12:39:41.721288 21359 leader_election.cc:288] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:41.721423 21364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:41.721812 21364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [term 1 LEADER]: Becoming Leader. State: Replica: 6c4176a63c444aa4bde90e4cd25b3e9d, State: Running, Role: LEADER
I20260927 12:39:41.722182 21364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } }
I20260927 12:39:41.722646 21359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:41.723022 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:41.708974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:56295 (local address 127.0.0.1:37011)
0927 12:39:41.709393 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:41.709410 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:41.709446 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:39:41.710101 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:41.710119 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:41.710494 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:41.710814 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:41.711700 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:41.712915 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:41.714015 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:41.714510 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:41.714634 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:41.717362 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:41.717369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:41.717389 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:41.717423 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:41.719419 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:41.719980 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:41.719987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:41.719994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:41.720177 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:41.720483 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:41.720487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:41.720489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:41.721030 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:41.721237 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:41.721828 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 12:39:41.722076 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260927 12:39:41.723860 21365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } } }
I20260927 12:39:41.723994 21365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:41.724198 21366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c4176a63c444aa4bde90e4cd25b3e9d. Latest consensus state: current_term: 1 leader_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c4176a63c444aa4bde90e4cd25b3e9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37011 } } }
I20260927 12:39:41.724306 21366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:41.724565 21373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:41.726722 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:41.727737 21373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:41.734215 21373 catalog_manager.cc:1347] Generated new cluster ID: 605cfa631ef14221b6b12f1604f5fbc7
I20260927 12:39:41.734334 21373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:41.739171 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:41.728843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:55439 (local address 127.0.0.1:37011)
0927 12:39:41.728952 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:41.728956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:41.728987 (+ 31us) server_negotiation.cc:407] Connection header received
0927 12:39:41.729024 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:41.729027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:41.729068 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:41.729129 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:41.729778 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:41.730524 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:41.732183 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:41.732540 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:41.732631 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:41.734584 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:41.734588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:41.734591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:41.734622 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:41.736520 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:41.737465 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:41.737473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:41.737475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:41.737683 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:41.738165 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:41.738169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:41.738170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:41.738426 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:41.738667 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:41.738876 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 12:39:41.738994 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:39:41.745577 21373 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:41.746822 21373 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:41.754408 21373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c4176a63c444aa4bde90e4cd25b3e9d: Generated new TSK 0
I20260927 12:39:41.755226 21373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:41.763392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21291
W20260927 12:39:41.768575 21276 connection.cc:535] client connection to 127.0.0.1:37011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37011): Connection reset by peer (error 104)
2026-09-27T12:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:41 dist-test-slave-tzgx krb5kdc[21390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:41 dist-test-slave-tzgx krb5kdc[21390](info): set up 2 sockets
Sep 27 12:39:41 dist-test-slave-tzgx krb5kdc[21390](info): commencing operation
krb5kdc: starting...
W20260927 12:39:42.744434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:42 dist-test-slave-tzgx krb5kdc[21390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:42.879977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:43.000061 21403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:43.000417 21403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:43.000476 21403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:43.004824 21403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:43.004913 21403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:43.004938 21403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:43.004962 21403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:43.004980 21403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:43.010978 21403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:43.013126 21403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:43.014317 21403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:43.024423 21411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:43.024407 21409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:43.024608 21403 server_base.cc:1034] running on GCE node
W20260927 12:39:43.024410 21408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:43.025557 21403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:43.027164 21403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:43.030537 21403 hybrid_clock.cc:648] HybridClock initialized: now 1790512783030447 us; error 88 us; skew 500 ppm
Sep 27 12:39:43 dist-test-slave-tzgx krb5kdc[21390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:43.038177 21403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:43.039866 21403 webserver.cc:460] Webserver started at http://127.0.0.1:37735/ using document root <none> and password file <none>
I20260927 12:39:43.040649 21403 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:43.040743 21403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:43.040973 21403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:43.043641 21403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "d60cc05df1c7409f8b44b103c2c02b12"
format_stamp: "Formatted at 2026-09-27 12:39:43 on dist-test-slave-tzgx"
server_key: "68574fa956ba7f2a2c881cd7face3b04"
server_key_iv: "afe2e1fde4b7191b54c12d6a92e6c5fe"
server_key_version: "encryptionkey@0"
I20260927 12:39:43.044418 21403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "d60cc05df1c7409f8b44b103c2c02b12"
format_stamp: "Formatted at 2026-09-27 12:39:43 on dist-test-slave-tzgx"
server_key: "68574fa956ba7f2a2c881cd7face3b04"
server_key_iv: "afe2e1fde4b7191b54c12d6a92e6c5fe"
server_key_version: "encryptionkey@0"
I20260927 12:39:43.048887 21403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:39:43.052490 21418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:43.054360 21403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 12:39:43.054790 21403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "d60cc05df1c7409f8b44b103c2c02b12"
format_stamp: "Formatted at 2026-09-27 12:39:43 on dist-test-slave-tzgx"
server_key: "68574fa956ba7f2a2c881cd7face3b04"
server_key_iv: "afe2e1fde4b7191b54c12d6a92e6c5fe"
server_key_version: "encryptionkey@0"
I20260927 12:39:43.055042 21403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:43.088549 21403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:43.092893 21403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:43.093122 21403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:43.103473 21403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36327
I20260927 12:39:43.103540 21470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36327 every 8 connection(s)
I20260927 12:39:43.104657 21403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:43.107669 21471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:43.108028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21403
I20260927 12:39:43.108187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:43.108479 392 external_mini_cluster.cc:1428] Setting key 427d65837c90550006a236fdd0e4112e
I20260927 12:39:43.114916 21471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12: Bootstrap starting.
I20260927 12:39:43.117818 21471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:43.118785 21471 log.cc:824] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:43.120579 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:43.109898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:38473 (local address 127.0.0.1:36327)
0927 12:39:43.110410 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:43.110426 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:43.110469 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:39:43.111293 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:43.111316 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:43.111721 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:43.112125 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:43.112581 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:43.113605 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:43.114567 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:43.115104 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:43.115262 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:43.116948 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:43.116953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:43.116969 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:43.118374 (+ 1405us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:43.118637 (+ 263us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:43.118717 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:43.118904 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:43.119138 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 12:39:43.119321 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":158,"threads_started":1}
I20260927 12:39:43.121546 21471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12: No bootstrap required, opened a new log
I20260927 12:39:43.126999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:43.127774 21471 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60cc05df1c7409f8b44b103c2c02b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36327 } }
I20260927 12:39:43.128118 21471 raft_consensus.cc:374] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:43.128187 21471 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60cc05df1c7409f8b44b103c2c02b12, State: Initialized, Role: FOLLOWER
I20260927 12:39:43.128683 21471 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60cc05df1c7409f8b44b103c2c02b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36327 } }
I20260927 12:39:43.128842 21471 raft_consensus.cc:388] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:43.128923 21471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:43.129029 21471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:43.130493 21471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60cc05df1c7409f8b44b103c2c02b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36327 } }
I20260927 12:39:43.131417 21471 leader_election.cc:302] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d60cc05df1c7409f8b44b103c2c02b12; no voters:
I20260927 12:39:43.131887 21471 leader_election.cc:288] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:43.131924 21481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:43.132234 21481 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [term 1 LEADER]: Becoming Leader. State: Replica: d60cc05df1c7409f8b44b103c2c02b12, State: Running, Role: LEADER
I20260927 12:39:43.132673 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60cc05df1c7409f8b44b103c2c02b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36327 } }
I20260927 12:39:43.134090 21471 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:43.135462 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d60cc05df1c7409f8b44b103c2c02b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60cc05df1c7409f8b44b103c2c02b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36327 } } }
I20260927 12:39:43.135632 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:43.135795 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:43.129646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:43555 (local address 127.0.0.1:36327)
0927 12:39:43.129800 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:43.129803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:43.130060 (+ 257us) server_negotiation.cc:407] Connection header received
0927 12:39:43.130154 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:43.130157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:43.130206 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:43.130283 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:43.130878 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:43.131734 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:43.133487 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:43.133868 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:43.134093 (+ 225us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:43.135222 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:43.135226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:43.135229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:43.135246 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:43.135292 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:43.135309 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:43.135400 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:43.135584 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 12:39:43.135623 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:39:43.136226 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60cc05df1c7409f8b44b103c2c02b12. Latest consensus state: current_term: 1 leader_uuid: "d60cc05df1c7409f8b44b103c2c02b12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d60cc05df1c7409f8b44b103c2c02b12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36327 } } }
I20260927 12:39:43.136345 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:43.136999 21490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:43.140991 21490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:43.147943 21490 catalog_manager.cc:1347] Generated new cluster ID: 0372ef6691f14199a2f245777ac09858
I20260927 12:39:43.148041 21490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:43.170629 21490 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:43.171752 21490 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:43.186007 21490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d60cc05df1c7409f8b44b103c2c02b12: Generated new TSK 0
I20260927 12:39:43.186873 21490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:43.201680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21403
W20260927 12:39:43.207235 21388 connection.cc:535] client connection to 127.0.0.1:36327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36327): Connection reset by peer (error 104)
2026-09-27T12:39:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:43 dist-test-slave-tzgx krb5kdc[21502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:43 dist-test-slave-tzgx krb5kdc[21502](info): set up 2 sockets
Sep 27 12:39:43 dist-test-slave-tzgx krb5kdc[21502](info): commencing operation
krb5kdc: starting...
W20260927 12:39:44.188060 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:44 dist-test-slave-tzgx krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:44.320604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:44.435494 21515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:44.435786 21515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:44.435835 21515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:44.439936 21515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:44.440021 21515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:44.440039 21515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:44.440055 21515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:44.440069 21515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:44.445744 21515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:44.446861 21515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:44.447988 21515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:44.456673 21520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:44.456645 21521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:44.456871 21515 server_base.cc:1034] running on GCE node
W20260927 12:39:44.456687 21523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:44.457533 21515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:44.458698 21515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:44.459872 21515 hybrid_clock.cc:648] HybridClock initialized: now 1790512784459852 us; error 34 us; skew 500 ppm
Sep 27 12:39:44 dist-test-slave-tzgx krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:44.463083 21515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:44.464134 21515 webserver.cc:460] Webserver started at http://127.0.0.1:46635/ using document root <none> and password file <none>
I20260927 12:39:44.464708 21515 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:44.464773 21515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:44.464977 21515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:44.467096 21515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "d71774f33fe041528fcdd46c548633bf"
format_stamp: "Formatted at 2026-09-27 12:39:44 on dist-test-slave-tzgx"
server_key: "84b85e63791ec32c14c73036ff9d2a40"
server_key_iv: "b4e6663480d803c110256ce35b08148c"
server_key_version: "encryptionkey@0"
I20260927 12:39:44.467672 21515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "d71774f33fe041528fcdd46c548633bf"
format_stamp: "Formatted at 2026-09-27 12:39:44 on dist-test-slave-tzgx"
server_key: "84b85e63791ec32c14c73036ff9d2a40"
server_key_iv: "b4e6663480d803c110256ce35b08148c"
server_key_version: "encryptionkey@0"
I20260927 12:39:44.471755 21515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 12:39:44.474718 21530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:44.476065 21515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:44.476227 21515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "d71774f33fe041528fcdd46c548633bf"
format_stamp: "Formatted at 2026-09-27 12:39:44 on dist-test-slave-tzgx"
server_key: "84b85e63791ec32c14c73036ff9d2a40"
server_key_iv: "b4e6663480d803c110256ce35b08148c"
server_key_version: "encryptionkey@0"
I20260927 12:39:44.476353 21515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:44.481220 21515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:44.485360 21515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:44.485595 21515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:44.493357 21515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37043
I20260927 12:39:44.493377 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37043 every 8 connection(s)
I20260927 12:39:44.494602 21515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:44.497705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21515
I20260927 12:39:44.497792 21583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:44.497860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:44.498119 392 external_mini_cluster.cc:1428] Setting key ae9274495334e9063eed1a1cd5b7006a
I20260927 12:39:44.504566 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:44.499579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45805 (local address 127.0.0.1:37043)
0927 12:39:44.500037 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:44.500054 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:44.500155 (+ 101us) server_negotiation.cc:407] Connection header received
0927 12:39:44.501112 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:44.501133 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:44.501467 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:44.501774 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:44.502896 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:44.502905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:44.502926 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:44.502949 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:44.503063 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:44.503095 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:44.503285 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:44.503487 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:39:44.503742 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260927 12:39:44.505441 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf: Bootstrap starting.
I20260927 12:39:44.507742 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:44.508128 21583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:44.508957 21583 log.cc:824] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:44.511375 21583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf: No bootstrap required, opened a new log
I20260927 12:39:44.512279 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:44.510411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:49355 (local address 127.0.0.1:37043)
0927 12:39:44.510552 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:44.510557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:44.510784 (+ 227us) server_negotiation.cc:407] Connection header received
0927 12:39:44.510920 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:44.510923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:44.510957 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:44.511030 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:44.511830 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:44.511834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:44.511836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:44.511847 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:44.511876 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:44.511887 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:44.511964 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:44.512064 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 12:39:44.512134 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:39:44.515583 21583 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71774f33fe041528fcdd46c548633bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260927 12:39:44.515880 21583 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:44.515959 21583 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71774f33fe041528fcdd46c548633bf, State: Initialized, Role: FOLLOWER
I20260927 12:39:44.516475 21583 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71774f33fe041528fcdd46c548633bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260927 12:39:44.516640 21583 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:44.516716 21583 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:44.516821 21583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:44.517961 21583 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71774f33fe041528fcdd46c548633bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260927 12:39:44.518316 21583 leader_election.cc:302] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71774f33fe041528fcdd46c548633bf; no voters:
I20260927 12:39:44.518731 21583 leader_election.cc:288] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:44.518850 21593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:44.519071 21593 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [term 1 LEADER]: Becoming Leader. State: Replica: d71774f33fe041528fcdd46c548633bf, State: Running, Role: LEADER
I20260927 12:39:44.519387 21593 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71774f33fe041528fcdd46c548633bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } }
I20260927 12:39:44.519958 21583 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:44.521076 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71774f33fe041528fcdd46c548633bf. Latest consensus state: current_term: 1 leader_uuid: "d71774f33fe041528fcdd46c548633bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71774f33fe041528fcdd46c548633bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260927 12:39:44.521224 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:44.521523 21594 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71774f33fe041528fcdd46c548633bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71774f33fe041528fcdd46c548633bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37043 } } }
I20260927 12:39:44.521628 21594 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:44.521602 21602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:44.525444 21602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:44.531533 21602 catalog_manager.cc:1347] Generated new cluster ID: 3e3c5cfec1754e869666786385da5d31
I20260927 12:39:44.531626 21602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:44.558710 21602 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:44.559103 21602 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:44.566119 21602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71774f33fe041528fcdd46c548633bf: Generated new TSK 0
I20260927 12:39:44.567099 21602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:44.579375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21515
2026-09-27T12:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:44 dist-test-slave-tzgx krb5kdc[21614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:44 dist-test-slave-tzgx krb5kdc[21614](info): set up 2 sockets
Sep 27 12:39:44 dist-test-slave-tzgx krb5kdc[21614](info): commencing operation
krb5kdc: starting...
W20260927 12:39:45.540169 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:45 dist-test-slave-tzgx krb5kdc[21614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:45.671223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:45.790706 21627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:45.791035 21627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:45.791107 21627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:45.795491 21627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:45.795590 21627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:45.795619 21627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:45.795641 21627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:45.795663 21627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:45.802119 21627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:45.803347 21627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:45.804471 21627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:39:45.813472 21627 server_base.cc:1034] running on GCE node
W20260927 12:39:45.813483 21635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:45.813391 21632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:45.813400 21633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:45.814471 21627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:45.815696 21627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:45.816910 21627 hybrid_clock.cc:648] HybridClock initialized: now 1790512785816861 us; error 64 us; skew 500 ppm
Sep 27 12:39:45 dist-test-slave-tzgx krb5kdc[21614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:45.819988 21627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:45.821060 21627 webserver.cc:460] Webserver started at http://127.0.0.1:35785/ using document root <none> and password file <none>
I20260927 12:39:45.821655 21627 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:45.821723 21627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:45.821928 21627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:45.824208 21627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "f0afb1d94f3240fa83e3486a13fa6173"
format_stamp: "Formatted at 2026-09-27 12:39:45 on dist-test-slave-tzgx"
server_key: "867df66a81b364c3d31a5b85d4a1babf"
server_key_iv: "94888dad6fd567985e587f109d2208e4"
server_key_version: "encryptionkey@0"
I20260927 12:39:45.824805 21627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0afb1d94f3240fa83e3486a13fa6173"
format_stamp: "Formatted at 2026-09-27 12:39:45 on dist-test-slave-tzgx"
server_key: "867df66a81b364c3d31a5b85d4a1babf"
server_key_iv: "94888dad6fd567985e587f109d2208e4"
server_key_version: "encryptionkey@0"
I20260927 12:39:45.828962 21627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:45.831669 21642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:45.833171 21627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:45.833310 21627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "f0afb1d94f3240fa83e3486a13fa6173"
format_stamp: "Formatted at 2026-09-27 12:39:45 on dist-test-slave-tzgx"
server_key: "867df66a81b364c3d31a5b85d4a1babf"
server_key_iv: "94888dad6fd567985e587f109d2208e4"
server_key_version: "encryptionkey@0"
I20260927 12:39:45.833446 21627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:45.861045 21627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:45.864804 21627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:45.864989 21627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:45.871976 21627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36465
I20260927 12:39:45.872066 21694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36465 every 8 connection(s)
I20260927 12:39:45.873061 21627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:45.876190 21695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:45.878800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21627
I20260927 12:39:45.878930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:45.879204 392 external_mini_cluster.cc:1428] Setting key ac57dc40ab994ee9f93071affe8b9095
I20260927 12:39:45.882823 21695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173: Bootstrap starting.
I20260927 12:39:45.885691 21695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:45.886528 21695 log.cc:824] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:45.889158 21695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173: No bootstrap required, opened a new log
Sep 27 12:39:45 dist-test-slave-tzgx krb5kdc[21614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:39:45.893651 21695 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0afb1d94f3240fa83e3486a13fa6173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260927 12:39:45.894068 21695 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:45.894129 21695 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0afb1d94f3240fa83e3486a13fa6173, State: Initialized, Role: FOLLOWER
I20260927 12:39:45.894851 21695 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0afb1d94f3240fa83e3486a13fa6173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260927 12:39:45.895040 21695 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:45.895109 21695 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:45.895208 21695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:45.896557 21695 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0afb1d94f3240fa83e3486a13fa6173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260927 12:39:45.897075 21695 leader_election.cc:302] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0afb1d94f3240fa83e3486a13fa6173; no voters:
I20260927 12:39:45.897393 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:45.880803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:42779 (local address 127.0.0.1:36465)
0927 12:39:45.881374 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:45.881394 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:45.881443 (+ 49us) server_negotiation.cc:407] Connection header received
0927 12:39:45.882289 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:45.882315 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:45.882829 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:45.883238 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:45.884042 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:45.885578 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:45.886635 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:45.887249 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:45.887381 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:45.890001 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:45.890009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:45.890029 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:45.890065 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:45.892998 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:45.893756 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:45.893764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:45.893770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:45.893954 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:45.894751 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:45.894755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:45.894757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:45.895365 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:45.895628 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:45.895898 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:39:45.896206 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":159,"threads_started":1}
I20260927 12:39:45.897476 21695 leader_election.cc:288] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:45.898311 21700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:45.899317 21695 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:45.898655 21700 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [term 1 LEADER]: Becoming Leader. State: Replica: f0afb1d94f3240fa83e3486a13fa6173, State: Running, Role: LEADER
I20260927 12:39:45.900287 21700 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0afb1d94f3240fa83e3486a13fa6173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260927 12:39:45.901777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:45.901885 21700 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0afb1d94f3240fa83e3486a13fa6173. Latest consensus state: current_term: 1 leader_uuid: "f0afb1d94f3240fa83e3486a13fa6173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0afb1d94f3240fa83e3486a13fa6173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260927 12:39:45.902019 21700 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:45.902375 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:45.905179 21701 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0afb1d94f3240fa83e3486a13fa6173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0afb1d94f3240fa83e3486a13fa6173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260927 12:39:45.905306 21701 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:45.908084 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:45.918679 21715 catalog_manager.cc:1347] Generated new cluster ID: 7e5096b4b1d1479f8a2205e2eaba8b50
I20260927 12:39:45.918764 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:45.919541 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:45.904520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:57529 (local address 127.0.0.1:36465)
0927 12:39:45.904633 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:45.904637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:45.907711 (+ 3074us) server_negotiation.cc:407] Connection header received
0927 12:39:45.908110 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:45.908115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:45.908175 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:45.908233 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:45.910179 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:45.910959 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:45.912174 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:45.912520 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:45.912605 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:45.914662 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:45.914667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:45.914669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:45.914704 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:45.917090 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:45.917797 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:45.917802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:45.917804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:45.917906 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:45.918323 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:45.918327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:45.918329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:45.918568 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:45.918689 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:45.919245 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 12:39:45.919369 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:39:45.932713 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:45.934211 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:45.949527 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0afb1d94f3240fa83e3486a13fa6173: Generated new TSK 0
I20260927 12:39:45.950589 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:45.956863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21627
W20260927 12:39:45.962249 21612 connection.cc:535] client connection to 127.0.0.1:36465 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36465): Connection reset by peer (error 104)
2026-09-27T12:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:46 dist-test-slave-tzgx krb5kdc[21726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:46 dist-test-slave-tzgx krb5kdc[21726](info): set up 2 sockets
Sep 27 12:39:46 dist-test-slave-tzgx krb5kdc[21726](info): commencing operation
krb5kdc: starting...
W20260927 12:39:46.940081 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:47 dist-test-slave-tzgx krb5kdc[21726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:47.068001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:47.182775 21739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:47.183089 21739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:47.183149 21739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:47.187636 21739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:47.187749 21739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:47.187780 21739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:47.187801 21739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:47.187820 21739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:47.194799 21739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:47.196008 21739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:47.197121 21739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:47.206097 21744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:47.206100 21747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:47.206118 21739 server_base.cc:1034] running on GCE node
W20260927 12:39:47.206100 21745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:47.207119 21739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:47.208457 21739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:47.209656 21739 hybrid_clock.cc:648] HybridClock initialized: now 1790512787209637 us; error 33 us; skew 500 ppm
Sep 27 12:39:47 dist-test-slave-tzgx krb5kdc[21726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:47.212780 21739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:47.213801 21739 webserver.cc:460] Webserver started at http://127.0.0.1:40471/ using document root <none> and password file <none>
I20260927 12:39:47.214413 21739 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:47.214478 21739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:47.214659 21739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:47.216934 21739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "06dba5fa47c4419bae40992a747f97df"
format_stamp: "Formatted at 2026-09-27 12:39:47 on dist-test-slave-tzgx"
server_key: "c271d7a26b63f8bfbaf9dcbf0889d211"
server_key_iv: "8dcc7bb841545b188b1dcc4695320652"
server_key_version: "encryptionkey@0"
I20260927 12:39:47.217484 21739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "06dba5fa47c4419bae40992a747f97df"
format_stamp: "Formatted at 2026-09-27 12:39:47 on dist-test-slave-tzgx"
server_key: "c271d7a26b63f8bfbaf9dcbf0889d211"
server_key_iv: "8dcc7bb841545b188b1dcc4695320652"
server_key_version: "encryptionkey@0"
I20260927 12:39:47.221593 21739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:47.224514 21754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:47.225946 21739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:47.226074 21739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "06dba5fa47c4419bae40992a747f97df"
format_stamp: "Formatted at 2026-09-27 12:39:47 on dist-test-slave-tzgx"
server_key: "c271d7a26b63f8bfbaf9dcbf0889d211"
server_key_iv: "8dcc7bb841545b188b1dcc4695320652"
server_key_version: "encryptionkey@0"
I20260927 12:39:47.226207 21739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:47.241726 21739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:47.245002 21739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:47.245195 21739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:47.252359 21739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41463
I20260927 12:39:47.252378 21806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41463 every 8 connection(s)
I20260927 12:39:47.253592 21739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:47.254940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21739
I20260927 12:39:47.255091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:47.255364 392 external_mini_cluster.cc:1428] Setting key e85bfd884149d29590d3f69522a3f83b
I20260927 12:39:47.257177 21807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:47.263127 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df: Bootstrap starting.
I20260927 12:39:47.264392 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:47.256810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:56561 (local address 127.0.0.1:41463)
0927 12:39:47.257329 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:47.257348 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:47.257387 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:39:47.258054 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:47.258071 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:47.258440 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:47.258780 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:47.259204 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:47.260278 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:47.261186 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:47.261660 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:47.261804 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:47.262652 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:47.262657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:47.262671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:47.263055 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:47.263110 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:47.263145 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:47.263262 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:47.263445 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 12:39:47.263577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":156,"threads_started":1}
I20260927 12:39:47.265775 21807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:47.266680 21807 log.cc:824] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:47.267556 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:47.268798 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df: No bootstrap required, opened a new log
I20260927 12:39:47.273432 21807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dba5fa47c4419bae40992a747f97df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41463 } }
I20260927 12:39:47.273749 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:47.269516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:41139 (local address 127.0.0.1:41463)
0927 12:39:47.269615 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:47.269619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:47.269807 (+ 188us) server_negotiation.cc:407] Connection header received
0927 12:39:47.269930 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:47.269932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:47.269972 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:47.270034 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:47.270305 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:47.270793 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:47.271914 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:47.272175 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:47.272252 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:47.273187 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:47.273190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:47.273192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:47.273206 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:47.273235 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:47.273245 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:47.273325 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:47.273553 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 12:39:47.273600 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 12:39:47.273761 21807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:47.273851 21807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06dba5fa47c4419bae40992a747f97df, State: Initialized, Role: FOLLOWER
I20260927 12:39:47.274446 21807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dba5fa47c4419bae40992a747f97df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41463 } }
I20260927 12:39:47.274637 21807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:47.274721 21807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:47.274837 21807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:47.275875 21807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dba5fa47c4419bae40992a747f97df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41463 } }
I20260927 12:39:47.276271 21807 leader_election.cc:302] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06dba5fa47c4419bae40992a747f97df; no voters:
I20260927 12:39:47.276578 21807 leader_election.cc:288] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:47.276700 21817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:47.276943 21817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [term 1 LEADER]: Becoming Leader. State: Replica: 06dba5fa47c4419bae40992a747f97df, State: Running, Role: LEADER
I20260927 12:39:47.277321 21817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dba5fa47c4419bae40992a747f97df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41463 } }
I20260927 12:39:47.277837 21807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:47.279201 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06dba5fa47c4419bae40992a747f97df. Latest consensus state: current_term: 1 leader_uuid: "06dba5fa47c4419bae40992a747f97df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dba5fa47c4419bae40992a747f97df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41463 } } }
I20260927 12:39:47.279368 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:47.279207 21818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06dba5fa47c4419bae40992a747f97df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dba5fa47c4419bae40992a747f97df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41463 } } }
I20260927 12:39:47.279456 21818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df [sys.catalog]: This master's current role is: LEADER
W20260927 12:39:47.283582 21831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:47.283696 21831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:47.283777 21832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:47.284731 21832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:47.290244 21832 catalog_manager.cc:1347] Generated new cluster ID: d7d2da9d35584fd5acf7498a25f268b3
I20260927 12:39:47.290334 21832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:47.343218 21832 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:47.344475 21832 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:47.349013 21832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06dba5fa47c4419bae40992a747f97df: Generated new TSK 0
I20260927 12:39:47.350172 21832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:47.359184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21739
W20260927 12:39:47.364310 21724 connection.cc:535] client connection to 127.0.0.1:41463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41463): Connection reset by peer (error 104)
2026-09-27T12:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:47 dist-test-slave-tzgx krb5kdc[21838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:47 dist-test-slave-tzgx krb5kdc[21838](info): set up 2 sockets
Sep 27 12:39:47 dist-test-slave-tzgx krb5kdc[21838](info): commencing operation
krb5kdc: starting...
W20260927 12:39:48.340130 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:48 dist-test-slave-tzgx krb5kdc[21838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:39:48.468050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:48.584161 21851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:48.584484 21851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:48.584545 21851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:48.588893 21851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:48.588979 21851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:48.589005 21851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:48.589028 21851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:48.589056 21851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:48.594772 21851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:48.595937 21851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:48.597054 21851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:39:48.606398 21851 server_base.cc:1034] running on GCE node
W20260927 12:39:48.606446 21859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:48.606488 21856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:48.606408 21857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:48.607394 21851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:48.608659 21851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:48.609864 21851 hybrid_clock.cc:648] HybridClock initialized: now 1790512788609832 us; error 48 us; skew 500 ppm
Sep 27 12:39:48 dist-test-slave-tzgx krb5kdc[21838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:48.617902 21851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:39:48.618956 21851 webserver.cc:460] Webserver started at http://127.0.0.1:39125/ using document root <none> and password file <none>
I20260927 12:39:48.619553 21851 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:48.619629 21851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:48.619848 21851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:48.622062 21851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "6a3c4b181efb4ba681b17d534775e91a"
format_stamp: "Formatted at 2026-09-27 12:39:48 on dist-test-slave-tzgx"
server_key: "946aaba2cda31b1c2a200e633ae7f248"
server_key_iv: "ff58bf024baabf504f9dbb05c4cc09f4"
server_key_version: "encryptionkey@0"
I20260927 12:39:48.622637 21851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a3c4b181efb4ba681b17d534775e91a"
format_stamp: "Formatted at 2026-09-27 12:39:48 on dist-test-slave-tzgx"
server_key: "946aaba2cda31b1c2a200e633ae7f248"
server_key_iv: "ff58bf024baabf504f9dbb05c4cc09f4"
server_key_version: "encryptionkey@0"
I20260927 12:39:48.626506 21851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:48.629122 21866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:48.630425 21851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:39:48.630549 21851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "6a3c4b181efb4ba681b17d534775e91a"
format_stamp: "Formatted at 2026-09-27 12:39:48 on dist-test-slave-tzgx"
server_key: "946aaba2cda31b1c2a200e633ae7f248"
server_key_iv: "ff58bf024baabf504f9dbb05c4cc09f4"
server_key_version: "encryptionkey@0"
I20260927 12:39:48.630697 21851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:48.635494 21851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:48.638650 21851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:48.638835 21851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:48.646188 21851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34475
I20260927 12:39:48.646253 21918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34475 every 8 connection(s)
I20260927 12:39:48.647162 21851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:48.650170 21919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:48.654994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21851
I20260927 12:39:48.655143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:48.655421 392 external_mini_cluster.cc:1428] Setting key be408188e7893136000a244910cdd862
I20260927 12:39:48.656740 21919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a: Bootstrap starting.
I20260927 12:39:48.659477 21919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:48.660395 21919 log.cc:824] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:48.661273 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:48.657006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:48821 (local address 127.0.0.1:34475)
0927 12:39:48.657470 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:48.657491 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:48.657532 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:39:48.658370 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:48.658393 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:48.658823 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:48.659192 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:48.659958 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:48.659964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:48.659977 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:48.659988 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:48.660048 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:48.660072 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:48.660209 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:48.660278 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 12:39:48.660435 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260927 12:39:48.662461 21919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a: No bootstrap required, opened a new log
I20260927 12:39:48.664209 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:48.666785 21919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3c4b181efb4ba681b17d534775e91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 12:39:48.667068 21919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:48.667158 21919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3c4b181efb4ba681b17d534775e91a, State: Initialized, Role: FOLLOWER
I20260927 12:39:48.667694 21919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3c4b181efb4ba681b17d534775e91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 12:39:48.667861 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:48.666209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:43539 (local address 127.0.0.1:34475)
0927 12:39:48.666323 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:48.666328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:48.666502 (+ 174us) server_negotiation.cc:407] Connection header received
0927 12:39:48.666642 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:48.666645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:48.666679 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:48.666741 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:48.667488 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:48.667492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:48.667494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:48.667506 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:48.667537 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:48.667547 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:48.667614 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:48.667713 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 12:39:48.667747 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:39:48.667943 21919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:48.668030 21919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:48.668161 21919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:48.669377 21919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3c4b181efb4ba681b17d534775e91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 12:39:48.669723 21919 leader_election.cc:302] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a3c4b181efb4ba681b17d534775e91a; no voters:
I20260927 12:39:48.670007 21919 leader_election.cc:288] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:48.670176 21929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:48.670400 21929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [term 1 LEADER]: Becoming Leader. State: Replica: 6a3c4b181efb4ba681b17d534775e91a, State: Running, Role: LEADER
I20260927 12:39:48.671132 21929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3c4b181efb4ba681b17d534775e91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 12:39:48.672554 21919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:48.673507 21931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a3c4b181efb4ba681b17d534775e91a. Latest consensus state: current_term: 1 leader_uuid: "6a3c4b181efb4ba681b17d534775e91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3c4b181efb4ba681b17d534775e91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } } }
I20260927 12:39:48.673666 21931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:48.673936 21930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a3c4b181efb4ba681b17d534775e91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3c4b181efb4ba681b17d534775e91a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } } }
I20260927 12:39:48.674031 21930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:48.674306 21938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:48.678280 21938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:48.684825 21938 catalog_manager.cc:1347] Generated new cluster ID: 8c1c100ff15d4b6ca453ae4c90cdd54e
I20260927 12:39:48.684911 21938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:48.699928 21938 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:48.700341 21938 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:48.707032 21938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a3c4b181efb4ba681b17d534775e91a: Generated new TSK 0
I20260927 12:39:48.708079 21938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:48.717260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21851
2026-09-27T12:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:48 dist-test-slave-tzgx krb5kdc[21950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:48 dist-test-slave-tzgx krb5kdc[21950](info): set up 2 sockets
Sep 27 12:39:48 dist-test-slave-tzgx krb5kdc[21950](info): commencing operation
krb5kdc: starting...
W20260927 12:39:49.692298 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:49 dist-test-slave-tzgx krb5kdc[21950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:49.824074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39641
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:49.947750 21963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:49.948078 21963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:49.948175 21963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:49.952517 21963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:49.952603 21963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:49.952632 21963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:49.952653 21963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:49.952673 21963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:49.958644 21963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:49.959909 21963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:49.960994 21963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:49.970980 21971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:49.971021 21969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:49.970986 21968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:49.972319 21963 server_base.cc:1034] running on GCE node
I20260927 12:39:49.973021 21963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:49.974295 21963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:49.975492 21963 hybrid_clock.cc:648] HybridClock initialized: now 1790512789975467 us; error 36 us; skew 500 ppm
Sep 27 12:39:49 dist-test-slave-tzgx krb5kdc[21950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:49.978606 21963 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:49.979902 21963 webserver.cc:460] Webserver started at http://127.0.98.62:35935/ using document root <none> and password file <none>
I20260927 12:39:49.980558 21963 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:49.980628 21963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:49.980808 21963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:49.983048 21963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "9cc23aeb9b4946e4ab40a8b705c62883"
format_stamp: "Formatted at 2026-09-27 12:39:49 on dist-test-slave-tzgx"
server_key: "20e88e77f7b1a15efd6a307246308c67"
server_key_iv: "ba34bdcac1e7d1f83fdd7861c2690361"
server_key_version: "encryptionkey@0"
I20260927 12:39:49.983704 21963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cc23aeb9b4946e4ab40a8b705c62883"
format_stamp: "Formatted at 2026-09-27 12:39:49 on dist-test-slave-tzgx"
server_key: "20e88e77f7b1a15efd6a307246308c67"
server_key_iv: "ba34bdcac1e7d1f83fdd7861c2690361"
server_key_version: "encryptionkey@0"
I20260927 12:39:49.988768 21963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 12:39:49.991546 21978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:49.992971 21963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:49.993127 21963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "9cc23aeb9b4946e4ab40a8b705c62883"
format_stamp: "Formatted at 2026-09-27 12:39:49 on dist-test-slave-tzgx"
server_key: "20e88e77f7b1a15efd6a307246308c67"
server_key_iv: "ba34bdcac1e7d1f83fdd7861c2690361"
server_key_version: "encryptionkey@0"
I20260927 12:39:49.993259 21963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:50.019579 21963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:50.022821 21963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:50.023011 21963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:50.030296 21963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39641
I20260927 12:39:50.030349 22030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39641 every 8 connection(s)
I20260927 12:39:50.031625 21963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:50.034670 22031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:50.041136 22031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883: Bootstrap starting.
I20260927 12:39:50.041567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 21963
I20260927 12:39:50.041699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:50.041960 392 external_mini_cluster.cc:1428] Setting key 0ac2a45ddd9b8b74d7401a586c1aa64d
I20260927 12:39:50.044126 22031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:50.044924 22031 log.cc:824] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:50.046936 22031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883: No bootstrap required, opened a new log
I20260927 12:39:50.050998 22031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260927 12:39:50.051273 22031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:50.051326 22031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc23aeb9b4946e4ab40a8b705c62883, State: Initialized, Role: FOLLOWER
Sep 27 12:39:50 dist-test-slave-tzgx krb5kdc[21950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:50.051808 22031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260927 12:39:50.051942 22031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:50.052011 22031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:50.052135 22031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:50.053350 22031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260927 12:39:50.053783 22031 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc23aeb9b4946e4ab40a8b705c62883; no voters:
I20260927 12:39:50.054181 22031 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:50.054307 22036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:50.054517 22036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [term 1 LEADER]: Becoming Leader. State: Replica: 9cc23aeb9b4946e4ab40a8b705c62883, State: Running, Role: LEADER
I20260927 12:39:50.054911 22036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } }
I20260927 12:39:50.055568 22031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:50.056766 22037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } } }
I20260927 12:39:50.056766 22038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc23aeb9b4946e4ab40a8b705c62883. Latest consensus state: current_term: 1 leader_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc23aeb9b4946e4ab40a8b705c62883" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39641 } } }
I20260927 12:39:50.056910 22037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:50.056910 22038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:50.057472 22045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:50.058627 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:50.043512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33223 (local address 127.0.98.62:39641)
0927 12:39:50.044044 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:50.044060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:50.044153 (+ 93us) server_negotiation.cc:407] Connection header received
0927 12:39:50.044888 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:50.044906 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:50.045308 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:50.045638 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:50.046505 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:50.047653 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:50.049027 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:50.049589 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:50.049769 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:50.052574 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:50.052580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:50.052598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:50.052637 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:50.054683 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:50.055378 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:50.055384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:50.055391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:50.055533 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:50.055863 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:50.055866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:50.055868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:50.056432 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:50.057422 (+ 990us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:50.057618 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 12:39:50.057863 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":147,"threads_started":1}
I20260927 12:39:50.060877 22045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:50.061887 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:50.066876 22045 catalog_manager.cc:1347] Generated new cluster ID: f7865c9a50984f49929c2e56a0ce045f
I20260927 12:39:50.066956 22045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:50.072620 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:50.063946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45377 (local address 127.0.98.62:39641)
0927 12:39:50.064052 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:50.064056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:50.064345 (+ 289us) server_negotiation.cc:407] Connection header received
0927 12:39:50.064482 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:50.064485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:50.064534 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:50.064619 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:50.065443 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:50.066650 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:50.067529 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:50.067776 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:50.067864 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:50.069278 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:50.069282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:50.069285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:50.069311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:50.070625 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:50.071274 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:50.071278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:50.071279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:50.071359 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:50.071703 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:50.071706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:50.071707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:50.071902 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:50.072038 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:50.072339 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 12:39:50.072440 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:39:50.109205 22045 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:50.110514 22045 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:50.116885 22045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc23aeb9b4946e4ab40a8b705c62883: Generated new TSK 0
I20260927 12:39:50.117799 22045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:50.132653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 21963
W20260927 12:39:50.137344 21948 connection.cc:535] client connection to 127.0.98.62:39641 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39641): Connection reset by peer (error 104)
2026-09-27T12:39:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:50 dist-test-slave-tzgx krb5kdc[22062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:50 dist-test-slave-tzgx krb5kdc[22062](info): set up 2 sockets
Sep 27 12:39:50 dist-test-slave-tzgx krb5kdc[22062](info): commencing operation
krb5kdc: starting...
W20260927 12:39:51.104151 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:51 dist-test-slave-tzgx krb5kdc[22062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:51.247558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:51.362577 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:51.362879 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:51.362921 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:51.367153 22075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:51.367244 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:51.367262 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:51.367277 22075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:51.367291 22075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:51.372856 22075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:51.373965 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:51.374989 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:51.383638 22083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:51.383649 22081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:51.383662 22080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:51.384006 22075 server_base.cc:1034] running on GCE node
I20260927 12:39:51.384617 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:51.385772 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:51.386950 22075 hybrid_clock.cc:648] HybridClock initialized: now 1790512791386931 us; error 32 us; skew 500 ppm
Sep 27 12:39:51 dist-test-slave-tzgx krb5kdc[22062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:51.389693 22075 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:51.390671 22075 webserver.cc:460] Webserver started at http://127.0.98.62:43987/ using document root <none> and password file <none>
I20260927 12:39:51.391218 22075 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:51.391276 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:51.391462 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:51.393667 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "b59ba088d06344ee9fd527f911fb4140"
format_stamp: "Formatted at 2026-09-27 12:39:51 on dist-test-slave-tzgx"
server_key: "01c214a930bf3d78a18be93ae97130b9"
server_key_iv: "35cf3ef00d1bf55dd090bc9e79d5ba9e"
server_key_version: "encryptionkey@0"
I20260927 12:39:51.394284 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "b59ba088d06344ee9fd527f911fb4140"
format_stamp: "Formatted at 2026-09-27 12:39:51 on dist-test-slave-tzgx"
server_key: "01c214a930bf3d78a18be93ae97130b9"
server_key_iv: "35cf3ef00d1bf55dd090bc9e79d5ba9e"
server_key_version: "encryptionkey@0"
I20260927 12:39:51.398378 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:51.401103 22090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:51.402464 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:39:51.402640 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "b59ba088d06344ee9fd527f911fb4140"
format_stamp: "Formatted at 2026-09-27 12:39:51 on dist-test-slave-tzgx"
server_key: "01c214a930bf3d78a18be93ae97130b9"
server_key_iv: "35cf3ef00d1bf55dd090bc9e79d5ba9e"
server_key_version: "encryptionkey@0"
I20260927 12:39:51.402765 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:51.419355 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:51.422555 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:51.422742 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:51.429770 22075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35679
I20260927 12:39:51.429796 22142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35679 every 8 connection(s)
I20260927 12:39:51.430748 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:51.433703 22143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:51.434561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22075
I20260927 12:39:51.434711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:51.435106 392 external_mini_cluster.cc:1428] Setting key 2be83e831a9517528ba1c310c35b1a93
I20260927 12:39:51.441363 22143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140: Bootstrap starting.
I20260927 12:39:51.443845 22143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:51.444519 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:51.436665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47695 (local address 127.0.98.62:35679)
0927 12:39:51.437186 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:51.437201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:51.437240 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:39:51.437908 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:51.437927 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:51.438310 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:51.438649 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:51.439096 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:51.440146 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:51.441093 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:51.441630 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:51.441789 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:51.442679 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:51.442684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:51.442698 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:51.443115 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:51.443179 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:51.443204 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:51.443323 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:51.443506 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 12:39:51.443655 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":156,"threads_started":1}
I20260927 12:39:51.444701 22143 log.cc:824] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:51.446859 22143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140: No bootstrap required, opened a new log
I20260927 12:39:51.447646 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:51.451112 22143 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ba088d06344ee9fd527f911fb4140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35679 } }
I20260927 12:39:51.451397 22143 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:51.451454 22143 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59ba088d06344ee9fd527f911fb4140, State: Initialized, Role: FOLLOWER
I20260927 12:39:51.451901 22143 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ba088d06344ee9fd527f911fb4140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35679 } }
I20260927 12:39:51.452042 22143 raft_consensus.cc:388] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:51.452153 22143 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:51.452266 22143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:51.453343 22143 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ba088d06344ee9fd527f911fb4140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35679 } }
I20260927 12:39:51.453763 22143 leader_election.cc:302] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59ba088d06344ee9fd527f911fb4140; no voters:
I20260927 12:39:51.453935 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:51.449755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56471 (local address 127.0.98.62:35679)
0927 12:39:51.449871 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:51.449875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:51.450093 (+ 218us) server_negotiation.cc:407] Connection header received
0927 12:39:51.450227 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:51.450230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:51.450273 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:51.450336 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:51.450664 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:51.451200 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:51.452161 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:51.452506 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:51.452618 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:51.453398 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:51.453402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:51.453405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:51.453421 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:51.453452 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:51.453465 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:51.453569 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:51.453706 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 12:39:51.453745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:39:51.454210 22143 leader_election.cc:288] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:51.454291 22153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:51.454505 22153 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [term 1 LEADER]: Becoming Leader. State: Replica: b59ba088d06344ee9fd527f911fb4140, State: Running, Role: LEADER
I20260927 12:39:51.454881 22153 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ba088d06344ee9fd527f911fb4140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35679 } }
I20260927 12:39:51.455590 22143 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:51.456277 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59ba088d06344ee9fd527f911fb4140. Latest consensus state: current_term: 1 leader_uuid: "b59ba088d06344ee9fd527f911fb4140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ba088d06344ee9fd527f911fb4140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35679 } } }
I20260927 12:39:51.456589 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:51.456821 22154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59ba088d06344ee9fd527f911fb4140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59ba088d06344ee9fd527f911fb4140" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35679 } } }
I20260927 12:39:51.456923 22154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:51.457166 22162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:51.460634 22162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:51.466216 22162 catalog_manager.cc:1347] Generated new cluster ID: 94b2d14cfce34356a981830724bdb339
I20260927 12:39:51.466383 22162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:51.498629 22162 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:51.499895 22162 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:51.505522 22162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b59ba088d06344ee9fd527f911fb4140: Generated new TSK 0
I20260927 12:39:51.506464 22162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:51.514372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22075
W20260927 12:39:51.520550 22060 connection.cc:535] client connection to 127.0.98.62:35679 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35679): Connection reset by peer (error 104)
2026-09-27T12:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:51 dist-test-slave-tzgx krb5kdc[22174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:51 dist-test-slave-tzgx krb5kdc[22174](info): set up 2 sockets
Sep 27 12:39:51 dist-test-slave-tzgx krb5kdc[22174](info): commencing operation
krb5kdc: starting...
W20260927 12:39:52.484009 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:52 dist-test-slave-tzgx krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:52.613130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:52.729871 22187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:52.730180 22187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:52.730226 22187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:52.734432 22187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:52.734509 22187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:52.734530 22187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:52.734547 22187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:52.734561 22187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:52.741236 22187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:52.742343 22187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:52.743340 22187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:52.752170 22193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:52.752363 22187 server_base.cc:1034] running on GCE node
W20260927 12:39:52.752168 22195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:52.752184 22192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:52.753084 22187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:52.754431 22187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:52.755618 22187 hybrid_clock.cc:648] HybridClock initialized: now 1790512792755600 us; error 39 us; skew 500 ppm
Sep 27 12:39:52 dist-test-slave-tzgx krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:52.758447 22187 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:52.759438 22187 webserver.cc:460] Webserver started at http://127.0.98.62:34515/ using document root <none> and password file <none>
I20260927 12:39:52.760004 22187 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:52.760074 22187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:52.760295 22187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:52.762441 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "12b99eec863149aab524682f0a93c1de"
format_stamp: "Formatted at 2026-09-27 12:39:52 on dist-test-slave-tzgx"
server_key: "e67021acb72e3a510c5bf7d6f2ca460e"
server_key_iv: "020d57d22be724dcbd02290e4cd709eb"
server_key_version: "encryptionkey@0"
I20260927 12:39:52.763010 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "12b99eec863149aab524682f0a93c1de"
format_stamp: "Formatted at 2026-09-27 12:39:52 on dist-test-slave-tzgx"
server_key: "e67021acb72e3a510c5bf7d6f2ca460e"
server_key_iv: "020d57d22be724dcbd02290e4cd709eb"
server_key_version: "encryptionkey@0"
I20260927 12:39:52.767177 22187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:39:52.770026 22202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:52.771514 22187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 12:39:52.771656 22187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "12b99eec863149aab524682f0a93c1de"
format_stamp: "Formatted at 2026-09-27 12:39:52 on dist-test-slave-tzgx"
server_key: "e67021acb72e3a510c5bf7d6f2ca460e"
server_key_iv: "020d57d22be724dcbd02290e4cd709eb"
server_key_version: "encryptionkey@0"
I20260927 12:39:52.771777 22187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:52.776662 22187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:52.779764 22187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:52.779955 22187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:52.786939 22187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41703
I20260927 12:39:52.786959 22254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41703 every 8 connection(s)
I20260927 12:39:52.787926 22187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:52.789564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22187
I20260927 12:39:52.789696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:52.790005 392 external_mini_cluster.cc:1428] Setting key cc5a0b869d04107b2671ddfcd8e06c24
I20260927 12:39:52.791106 22255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:52.796553 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:52.791525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38801 (local address 127.0.98.62:41703)
0927 12:39:52.792006 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:52.792023 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:52.792314 (+ 291us) server_negotiation.cc:407] Connection header received
0927 12:39:52.793086 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:52.793105 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:52.793838 (+ 733us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:52.794220 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:52.795195 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:52.795200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:52.795214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:52.795226 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:52.795288 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:52.795308 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:52.795449 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:52.795541 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 12:39:52.795701 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260927 12:39:52.797791 22255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de: Bootstrap starting.
I20260927 12:39:52.799775 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:52.800550 22255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:52.801533 22255 log.cc:824] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:52.804015 22255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de: No bootstrap required, opened a new log
I20260927 12:39:52.804211 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:52.802115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34619 (local address 127.0.98.62:41703)
0927 12:39:52.802226 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:52.802231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:52.802390 (+ 159us) server_negotiation.cc:407] Connection header received
0927 12:39:52.802643 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:52.802648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:52.802690 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:52.802802 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:52.803735 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:52.803738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:52.803740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:52.803751 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:52.803785 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:52.803796 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:52.803904 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:52.803980 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 12:39:52.804023 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 12:39:52.809398 22255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b99eec863149aab524682f0a93c1de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41703 } }
I20260927 12:39:52.809846 22255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:52.809924 22255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b99eec863149aab524682f0a93c1de, State: Initialized, Role: FOLLOWER
I20260927 12:39:52.810381 22255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b99eec863149aab524682f0a93c1de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41703 } }
I20260927 12:39:52.810550 22255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:52.810604 22255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:52.810679 22255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:52.811650 22255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b99eec863149aab524682f0a93c1de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41703 } }
I20260927 12:39:52.812021 22255 leader_election.cc:302] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b99eec863149aab524682f0a93c1de; no voters:
I20260927 12:39:52.812388 22255 leader_election.cc:288] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:52.812562 22265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:52.812776 22265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [term 1 LEADER]: Becoming Leader. State: Replica: 12b99eec863149aab524682f0a93c1de, State: Running, Role: LEADER
I20260927 12:39:52.813113 22265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b99eec863149aab524682f0a93c1de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41703 } }
I20260927 12:39:52.813727 22255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:52.814850 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b99eec863149aab524682f0a93c1de. Latest consensus state: current_term: 1 leader_uuid: "12b99eec863149aab524682f0a93c1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b99eec863149aab524682f0a93c1de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41703 } } }
I20260927 12:39:52.814996 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:52.815182 22266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b99eec863149aab524682f0a93c1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b99eec863149aab524682f0a93c1de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41703 } } }
I20260927 12:39:52.815511 22266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:52.815582 22276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:52.818569 22276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:52.824484 22276 catalog_manager.cc:1347] Generated new cluster ID: 0635d98a7c5146f08aa0e53caf2d4ed6
I20260927 12:39:52.824585 22276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:52.843516 22276 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:52.843923 22276 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:52.851281 22276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12b99eec863149aab524682f0a93c1de: Generated new TSK 0
I20260927 12:39:52.852264 22276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:52.858701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22187
2026-09-27T12:39:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:52 dist-test-slave-tzgx krb5kdc[22286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:52 dist-test-slave-tzgx krb5kdc[22286](info): set up 2 sockets
Sep 27 12:39:52 dist-test-slave-tzgx krb5kdc[22286](info): commencing operation
krb5kdc: starting...
W20260927 12:39:53.844148 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:53 dist-test-slave-tzgx krb5kdc[22286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:53.981688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38385
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:54.101707 22299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:54.102033 22299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:54.102079 22299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:54.106431 22299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:54.106526 22299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:54.106545 22299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:54.106561 22299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:54.106575 22299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:54.112888 22299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:54.114120 22299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:54.115362 22299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:39:54.125411 22299 server_base.cc:1034] running on GCE node
W20260927 12:39:54.125495 22304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:54.125506 22305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:54.125658 22307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:54.126446 22299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:54.127737 22299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:54.128978 22299 hybrid_clock.cc:648] HybridClock initialized: now 1790512794128942 us; error 54 us; skew 500 ppm
Sep 27 12:39:54 dist-test-slave-tzgx krb5kdc[22286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:54.143745 22299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:54.144795 22299 webserver.cc:460] Webserver started at http://127.0.98.62:46379/ using document root <none> and password file <none>
I20260927 12:39:54.145347 22299 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:54.145413 22299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:54.145581 22299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:54.147703 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "d85010601b9d4629828c1d7b93b77cb0"
format_stamp: "Formatted at 2026-09-27 12:39:54 on dist-test-slave-tzgx"
server_key: "f64cca8f98cf3009c5f828e36820e26e"
server_key_iv: "ef749af29068ec08fcb7f36db61cad18"
server_key_version: "encryptionkey@0"
I20260927 12:39:54.148301 22299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "d85010601b9d4629828c1d7b93b77cb0"
format_stamp: "Formatted at 2026-09-27 12:39:54 on dist-test-slave-tzgx"
server_key: "f64cca8f98cf3009c5f828e36820e26e"
server_key_iv: "ef749af29068ec08fcb7f36db61cad18"
server_key_version: "encryptionkey@0"
I20260927 12:39:54.152323 22299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:39:54.155066 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:54.156697 22299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:54.156836 22299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "d85010601b9d4629828c1d7b93b77cb0"
format_stamp: "Formatted at 2026-09-27 12:39:54 on dist-test-slave-tzgx"
server_key: "f64cca8f98cf3009c5f828e36820e26e"
server_key_iv: "ef749af29068ec08fcb7f36db61cad18"
server_key_version: "encryptionkey@0"
I20260927 12:39:54.156952 22299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:54.168798 22299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:54.172652 22299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:54.172829 22299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:54.180445 22299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38385
I20260927 12:39:54.180464 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38385 every 8 connection(s)
I20260927 12:39:54.181485 22299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:54.184850 22367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:54.189494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22299
I20260927 12:39:54.189656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:54.189937 392 external_mini_cluster.cc:1428] Setting key dc66e0a5b2e51a23efd202c9420ac844
I20260927 12:39:54.192410 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0: Bootstrap starting.
I20260927 12:39:54.194952 22367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:54.195879 22367 log.cc:824] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:54.198148 22367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0: No bootstrap required, opened a new log
Sep 27 12:39:54 dist-test-slave-tzgx krb5kdc[22286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:54.203171 22367 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85010601b9d4629828c1d7b93b77cb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38385 } }
I20260927 12:39:54.203449 22367 raft_consensus.cc:374] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:54.203493 22367 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85010601b9d4629828c1d7b93b77cb0, State: Initialized, Role: FOLLOWER
I20260927 12:39:54.204010 22367 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85010601b9d4629828c1d7b93b77cb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38385 } }
I20260927 12:39:54.204207 22367 raft_consensus.cc:388] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:54.204330 22367 raft_consensus.cc:482] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:54.204430 22367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:54.205720 22367 raft_consensus.cc:504] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85010601b9d4629828c1d7b93b77cb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38385 } }
I20260927 12:39:54.206125 22367 leader_election.cc:302] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85010601b9d4629828c1d7b93b77cb0; no voters:
I20260927 12:39:54.206436 22367 leader_election.cc:288] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:54.206569 22372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:54.206796 22372 raft_consensus.cc:686] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [term 1 LEADER]: Becoming Leader. State: Replica: d85010601b9d4629828c1d7b93b77cb0, State: Running, Role: LEADER
I20260927 12:39:54.207175 22372 consensus_queue.cc:237] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85010601b9d4629828c1d7b93b77cb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38385 } }
I20260927 12:39:54.207329 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:54.191648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56709 (local address 127.0.98.62:38385)
0927 12:39:54.192211 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:54.192228 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:54.192268 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:39:54.193019 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:54.193038 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:54.193418 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:54.194186 (+ 768us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:54.195192 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:54.196627 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:54.197563 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:54.198252 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:54.198425 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:54.201337 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:54.201344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:54.201363 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:54.201397 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:54.203684 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:54.204288 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:54.204296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:54.204303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:54.204488 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:54.204821 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:54.204824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:54.204826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:54.205300 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:54.205472 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:54.205839 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 12:39:54.206216 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":174,"threads_started":1}
I20260927 12:39:54.207706 22367 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:54.208523 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d85010601b9d4629828c1d7b93b77cb0. Latest consensus state: current_term: 1 leader_uuid: "d85010601b9d4629828c1d7b93b77cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85010601b9d4629828c1d7b93b77cb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38385 } } }
I20260927 12:39:54.208698 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:54.208925 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d85010601b9d4629828c1d7b93b77cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85010601b9d4629828c1d7b93b77cb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38385 } } }
I20260927 12:39:54.209030 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:54.209105 22381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:54.210762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:54.212939 22381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:54.219123 22381 catalog_manager.cc:1347] Generated new cluster ID: 33a0ada9926f47a4a5a6092cdd6efb28
I20260927 12:39:54.219208 22381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:54.225302 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:54.212897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49987 (local address 127.0.98.62:38385)
0927 12:39:54.213004 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:54.213009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:54.215537 (+ 2528us) server_negotiation.cc:407] Connection header received
0927 12:39:54.215670 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:54.215673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:54.215726 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:54.215813 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:54.216786 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:54.217674 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:54.219250 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:54.219613 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:54.219699 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:54.221706 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:54.221711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:54.221713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:54.221745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:54.223344 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:54.223854 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:54.223858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:54.223859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:54.223939 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:54.224280 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:54.224284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:54.224286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:54.224524 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:54.224629 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:54.225003 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 12:39:54.225130 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:39:54.231269 22381 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:54.232668 22381 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:54.245713 22381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d85010601b9d4629828c1d7b93b77cb0: Generated new TSK 0
I20260927 12:39:54.246490 22381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:54.254935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22299
2026-09-27T12:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:54 dist-test-slave-tzgx krb5kdc[22398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:54 dist-test-slave-tzgx krb5kdc[22398](info): set up 2 sockets
Sep 27 12:39:54 dist-test-slave-tzgx krb5kdc[22398](info): commencing operation
krb5kdc: starting...
W20260927 12:39:55.244328 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:55 dist-test-slave-tzgx krb5kdc[22398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:55.377317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:55.489343 22411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:55.489651 22411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:55.489696 22411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:55.493963 22411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:55.494043 22411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:55.494063 22411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:55.494079 22411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:55.494094 22411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:55.499758 22411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:55.500952 22411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:55.501998 22411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:55.510893 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:55.510900 22417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:55.510898 22416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:55.511116 22411 server_base.cc:1034] running on GCE node
I20260927 12:39:55.511835 22411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:55.513010 22411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:55.514185 22411 hybrid_clock.cc:648] HybridClock initialized: now 1790512795514164 us; error 33 us; skew 500 ppm
Sep 27 12:39:55 dist-test-slave-tzgx krb5kdc[22398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:55.517007 22411 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:55.517961 22411 webserver.cc:460] Webserver started at http://127.0.98.62:34231/ using document root <none> and password file <none>
I20260927 12:39:55.518519 22411 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:55.518589 22411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:55.518805 22411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:55.520943 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "b685c2174a704c6a80a4e975c543b856"
format_stamp: "Formatted at 2026-09-27 12:39:55 on dist-test-slave-tzgx"
server_key: "73f7273c29674d09426e59795ae9b443"
server_key_iv: "0ff68d4b84d4fee507048d2bcffc33cb"
server_key_version: "encryptionkey@0"
I20260927 12:39:55.521497 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "b685c2174a704c6a80a4e975c543b856"
format_stamp: "Formatted at 2026-09-27 12:39:55 on dist-test-slave-tzgx"
server_key: "73f7273c29674d09426e59795ae9b443"
server_key_iv: "0ff68d4b84d4fee507048d2bcffc33cb"
server_key_version: "encryptionkey@0"
I20260927 12:39:55.525480 22411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:39:55.528018 22426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:55.529275 22411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:39:55.529418 22411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "b685c2174a704c6a80a4e975c543b856"
format_stamp: "Formatted at 2026-09-27 12:39:55 on dist-test-slave-tzgx"
server_key: "73f7273c29674d09426e59795ae9b443"
server_key_iv: "0ff68d4b84d4fee507048d2bcffc33cb"
server_key_version: "encryptionkey@0"
I20260927 12:39:55.529529 22411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:55.544008 22411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:55.547243 22411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:55.547456 22411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:55.554293 22411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34491
I20260927 12:39:55.554438 22478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34491 every 8 connection(s)
I20260927 12:39:55.555490 22411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:55.558532 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:55.564152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22411
I20260927 12:39:55.564314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:55.564599 392 external_mini_cluster.cc:1428] Setting key 59dd0d16034d67236844735370c39e69
I20260927 12:39:55.564836 22479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856: Bootstrap starting.
I20260927 12:39:55.567474 22479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:55.568286 22479 log.cc:824] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:55.570549 22479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856: No bootstrap required, opened a new log
I20260927 12:39:55.574409 22479 raft_consensus.cc:348] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b685c2174a704c6a80a4e975c543b856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260927 12:39:55.574729 22479 raft_consensus.cc:374] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:55.574787 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:55.566021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52491 (local address 127.0.98.62:34491)
0927 12:39:55.566555 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:55.566572 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:55.566614 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:39:55.567384 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:55.567404 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:55.567842 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:55.568237 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:55.568705 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:55.569888 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:55.571310 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:55.571855 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:55.572021 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:55.572862 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:55.572868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:55.572883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:55.573308 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:55.573370 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:55.573395 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:55.573487 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:55.573637 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 12:39:55.573879 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":171,"threads_started":1}
I20260927 12:39:55.574792 22479 raft_consensus.cc:729] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b685c2174a704c6a80a4e975c543b856, State: Initialized, Role: FOLLOWER
I20260927 12:39:55.575382 22479 consensus_queue.cc:260] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b685c2174a704c6a80a4e975c543b856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260927 12:39:55.575553 22479 raft_consensus.cc:388] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:55.575613 22479 raft_consensus.cc:482] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:55.575701 22479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:55.576766 22479 raft_consensus.cc:504] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b685c2174a704c6a80a4e975c543b856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260927 12:39:55.577122 22479 leader_election.cc:302] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b685c2174a704c6a80a4e975c543b856; no voters:
I20260927 12:39:55.577485 22479 leader_election.cc:288] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:55.577560 22484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:55.577750 22484 raft_consensus.cc:686] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [term 1 LEADER]: Becoming Leader. State: Replica: b685c2174a704c6a80a4e975c543b856, State: Running, Role: LEADER
I20260927 12:39:55.578086 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:55.578832 22479 sys_catalog.cc:564] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:55.580135 22484 consensus_queue.cc:237] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b685c2174a704c6a80a4e975c543b856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } }
I20260927 12:39:55.581863 22485 sys_catalog.cc:455] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b685c2174a704c6a80a4e975c543b856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b685c2174a704c6a80a4e975c543b856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } } }
I20260927 12:39:55.582024 22486 sys_catalog.cc:455] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b685c2174a704c6a80a4e975c543b856. Latest consensus state: current_term: 1 leader_uuid: "b685c2174a704c6a80a4e975c543b856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b685c2174a704c6a80a4e975c543b856" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34491 } } }
I20260927 12:39:55.582077 22486 sys_catalog.cc:458] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:55.582082 22485 sys_catalog.cc:458] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:55.582432 22504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:55.584743 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:55.580718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49353 (local address 127.0.98.62:34491)
0927 12:39:55.580823 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:55.580827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:55.580840 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:39:55.580873 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:55.580875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:55.580911 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:55.580978 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:55.581248 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:55.582001 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:55.583106 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:55.583413 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:55.583511 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:55.584319 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:55.584323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:55.584326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:55.584341 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:55.584377 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:55.584388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:55.584449 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:55.584544 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 12:39:55.584597 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:39:55.585667 22504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:55.591761 22504 catalog_manager.cc:1347] Generated new cluster ID: 26e695c92ab947b28195e8503646e2b1
I20260927 12:39:55.591857 22504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:55.613741 22504 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:55.615120 22504 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:55.621390 22504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b685c2174a704c6a80a4e975c543b856: Generated new TSK 0
I20260927 12:39:55.622136 22504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:55.632042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22411
2026-09-27T12:39:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:55 dist-test-slave-tzgx krb5kdc[22510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:55 dist-test-slave-tzgx krb5kdc[22510](info): set up 2 sockets
Sep 27 12:39:55 dist-test-slave-tzgx krb5kdc[22510](info): commencing operation
krb5kdc: starting...
W20260927 12:39:56.612272 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:56 dist-test-slave-tzgx krb5kdc[22510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:56.748531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45929
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:56.868932 22523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:56.869231 22523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:56.869275 22523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:56.873855 22523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:56.873975 22523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:56.873996 22523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:56.874012 22523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:56.874027 22523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:56.880324 22523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:56.881484 22523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:56.882560 22523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:56.891746 22529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:56.891960 22523 server_base.cc:1034] running on GCE node
W20260927 12:39:56.891881 22528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:56.891901 22531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:56.892776 22523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:56.894016 22523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:56.895262 22523 hybrid_clock.cc:648] HybridClock initialized: now 1790512796895257 us; error 83 us; skew 500 ppm
Sep 27 12:39:56 dist-test-slave-tzgx krb5kdc[22510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:56.899287 22523 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:56.900523 22523 webserver.cc:460] Webserver started at http://127.0.98.62:38691/ using document root <none> and password file <none>
I20260927 12:39:56.901106 22523 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:56.901177 22523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:56.901386 22523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:56.903610 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "784754c78ac046c9bfcf5dd6bbd106c2"
format_stamp: "Formatted at 2026-09-27 12:39:56 on dist-test-slave-tzgx"
server_key: "e151c62b47e03248a6ea1749744478e3"
server_key_iv: "517d41ed67a0f82e6f3ec73c09e17bc9"
server_key_version: "encryptionkey@0"
I20260927 12:39:56.904222 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "784754c78ac046c9bfcf5dd6bbd106c2"
format_stamp: "Formatted at 2026-09-27 12:39:56 on dist-test-slave-tzgx"
server_key: "e151c62b47e03248a6ea1749744478e3"
server_key_iv: "517d41ed67a0f82e6f3ec73c09e17bc9"
server_key_version: "encryptionkey@0"
I20260927 12:39:56.908710 22523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:39:56.911468 22538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:56.913416 22523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:39:56.913570 22523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "784754c78ac046c9bfcf5dd6bbd106c2"
format_stamp: "Formatted at 2026-09-27 12:39:56 on dist-test-slave-tzgx"
server_key: "e151c62b47e03248a6ea1749744478e3"
server_key_iv: "517d41ed67a0f82e6f3ec73c09e17bc9"
server_key_version: "encryptionkey@0"
I20260927 12:39:56.913738 22523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:56.918956 22523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:56.923051 22523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:56.923272 22523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:56.930588 22523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45929
I20260927 12:39:56.930604 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45929 every 8 connection(s)
I20260927 12:39:56.931588 22523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:56.934669 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:56.936066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22523
I20260927 12:39:56.936266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:56.936604 392 external_mini_cluster.cc:1428] Setting key cb7bec016dca18628cc03d635e6e52c9
I20260927 12:39:56.941857 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2: Bootstrap starting.
I20260927 12:39:56.942894 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:56.938348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39015 (local address 127.0.98.62:45929)
0927 12:39:56.938988 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:56.939006 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:56.939048 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:39:56.939782 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:56.939801 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:56.940180 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:56.940630 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:56.941525 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:56.941530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:56.941543 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:56.941557 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:56.941620 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:56.941641 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:56.941782 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:56.941896 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 12:39:56.942053 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":191,"threads_started":1}
I20260927 12:39:56.945027 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:56.945837 22591 log.cc:824] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:56.945953 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:56.948021 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2: No bootstrap required, opened a new log
I20260927 12:39:56.949875 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:56.948222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38637 (local address 127.0.98.62:45929)
0927 12:39:56.948347 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:56.948352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:56.948530 (+ 178us) server_negotiation.cc:407] Connection header received
0927 12:39:56.948662 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:56.948665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:56.948703 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:56.948771 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:56.949488 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:56.949492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:56.949495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:39:56.949507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:56.949537 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:39:56.949548 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:56.949626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:56.949699 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 12:39:56.949741 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:39:56.952194 22591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45929 } }
I20260927 12:39:56.952509 22591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:39:56.952571 22591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784754c78ac046c9bfcf5dd6bbd106c2, State: Initialized, Role: FOLLOWER
I20260927 12:39:56.953049 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45929 } }
I20260927 12:39:56.953202 22591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:39:56.953262 22591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:39:56.953341 22591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:56.954290 22591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45929 } }
I20260927 12:39:56.954620 22591 leader_election.cc:302] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 784754c78ac046c9bfcf5dd6bbd106c2; no voters:
I20260927 12:39:56.954921 22591 leader_election.cc:288] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:39:56.955052 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:56.955331 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [term 1 LEADER]: Becoming Leader. State: Replica: 784754c78ac046c9bfcf5dd6bbd106c2, State: Running, Role: LEADER
I20260927 12:39:56.955669 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45929 } }
I20260927 12:39:56.956327 22591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:56.957705 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 784754c78ac046c9bfcf5dd6bbd106c2. Latest consensus state: current_term: 1 leader_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45929 } } }
I20260927 12:39:56.957655 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "784754c78ac046c9bfcf5dd6bbd106c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45929 } } }
I20260927 12:39:56.957818 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:56.957818 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2 [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:56.958258 22610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:56.961644 22610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:56.969014 22610 catalog_manager.cc:1347] Generated new cluster ID: ca9f5a27c0b04abc8285f43def76882b
I20260927 12:39:56.969300 22610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:56.987970 22610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:56.988409 22610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:56.994315 22610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 784754c78ac046c9bfcf5dd6bbd106c2: Generated new TSK 0
I20260927 12:39:56.995093 22610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:39:57.003705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22523
2026-09-27T12:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1372 ms)
[----------] 18 tests from AuthTokenIssuingTest (24846 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:39:57 dist-test-slave-tzgx krb5kdc[22622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:39:57 dist-test-slave-tzgx krb5kdc[22622](info): set up 2 sockets
Sep 27 12:39:57 dist-test-slave-tzgx krb5kdc[22622](info): commencing operation
krb5kdc: starting...
W20260927 12:39:57.980240 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:39:58.104887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43903,127.0.98.61:40657,127.0.98.60:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:39:58.221809 22635 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:39:58.222129 22635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:58.222162 22635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:58.222201 22635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:58.226470 22635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:58.226548 22635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:58.226567 22635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:58.226583 22635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:58.226598 22635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:58.232417 22635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43903,127.0.98.61:40657,127.0.98.60:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:58.233722 22635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:58.234802 22635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:58.243541 22640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:58.243661 22641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:58.243662 22643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:58.243911 22635 server_base.cc:1034] running on GCE node
I20260927 12:39:58.244541 22635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:58.246260 22635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:58.247475 22635 hybrid_clock.cc:648] HybridClock initialized: now 1790512798247450 us; error 45 us; skew 500 ppm
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:58.250538 22635 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:39:58.251540 22635 webserver.cc:460] Webserver started at http://127.0.98.62:37543/ using document root <none> and password file <none>
I20260927 12:39:58.252128 22635 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:58.252202 22635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:58.252408 22635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:58.254572 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "1a74f2975ca348c591a953dc6f5f9e4b"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "27d6d6927ff1924a59e8d03b4dbc26c6"
server_key_iv: "b28398ce9d8329e87ee65808406e31ba"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.255157 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a74f2975ca348c591a953dc6f5f9e4b"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "27d6d6927ff1924a59e8d03b4dbc26c6"
server_key_iv: "b28398ce9d8329e87ee65808406e31ba"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.259670 22635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:39:58.262539 22650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:58.263934 22635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:58.264066 22635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "1a74f2975ca348c591a953dc6f5f9e4b"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "27d6d6927ff1924a59e8d03b4dbc26c6"
server_key_iv: "b28398ce9d8329e87ee65808406e31ba"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.264281 22635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:58.280370 22635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:58.284080 22635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:58.284315 22635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:58.292006 22635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43903
I20260927 12:39:58.292389 22702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43903 every 8 connection(s)
I20260927 12:39:58.293422 22635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:39:58.296643 22703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:58.300642 22703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:39:58.302248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22635
I20260927 12:39:58.302366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:39:58.302630 392 external_mini_cluster.cc:1428] Setting key 0dfcfcb855dbb86073c2fa1167960cec
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:58.320406 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.303197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43903 (local address 127.0.0.1:41426)
0927 12:39:58.303749 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.303806 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.304784 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.305507 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.305513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.305964 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.306319 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.306340 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.307961 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.307964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.308812 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.308822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.309743 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.309746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.309886 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.310506 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.310524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.314936 (+ 4412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.316954 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.316963 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.316975 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.317208 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.317522 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.317525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.317527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.317634 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.318532 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.318539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.318693 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.318859 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.319288 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":103,"thread_start_us":166,"threads_started":1}
I20260927 12:39:58.320406 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.304170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.0.98.62:43903)
0927 12:39:58.304478 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.304493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.304542 (+ 49us) server_negotiation.cc:407] Connection header received
0927 12:39:58.305100 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.305106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.305308 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.305464 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.306525 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.307803 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.308944 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.309561 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.309746 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.315122 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.315128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.315141 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.315167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.316828 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.317329 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.317335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.317336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.317440 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.317720 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.317723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.317724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.318390 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.318554 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.318957 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.319226 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260927 12:39:58.323300 22703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:39:58.324132 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.323808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33318)
0927 12:39:58.323932 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.324019 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 12:39:58.324378 22653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
I20260927 12:39:58.325307 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.325067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33332)
0927 12:39:58.325181 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.325229 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:39:58.325486 22703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:39:58.331854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40657
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43903,127.0.98.61:40657,127.0.98.60:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:39:58.374771 22703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } attempt: 1
I20260927 12:39:58.375600 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.375322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33342)
0927 12:39:58.375466 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.375524 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 12:39:58.376513 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.376269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33356)
0927 12:39:58.376388 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.376445 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:39:58.376632 22703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
I20260927 12:39:58.444793 22703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } attempt: 2
I20260927 12:39:58.445742 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.445392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33360)
0927 12:39:58.445573 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.445647 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 12:39:58.446936 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.446632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33368)
0927 12:39:58.446791 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.446854 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 12:39:58.447141 22703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40657: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40657: connect: Connection refused (error 111)
W20260927 12:39:58.450845 22710 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:39:58.451186 22710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:58.451238 22710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:58.451301 22710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:39:58.455631 22710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:58.455719 22710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:58.455747 22710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:58.455767 22710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:58.455788 22710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:58.461473 22710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43903,127.0.98.61:40657,127.0.98.60:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:58.462630 22710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:58.463649 22710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:58.472383 22718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:58.472368 22716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:58.472368 22715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:58.472668 22710 server_base.cc:1034] running on GCE node
I20260927 12:39:58.473215 22710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:58.474390 22710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:58.475572 22710 hybrid_clock.cc:648] HybridClock initialized: now 1790512798475559 us; error 28 us; skew 500 ppm
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:58.478487 22710 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:39:58.479461 22710 webserver.cc:460] Webserver started at http://127.0.98.61:33815/ using document root <none> and password file <none>
I20260927 12:39:58.479986 22710 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:58.480046 22710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:58.480280 22710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:58.482776 22710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data/instance:
uuid: "3b05d5a6c8a8485d87f873e3999cccd7"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "a6108f5ef19e464c426c1c828b490bc4"
server_key_iv: "22671ff71ca8a1c05d066ec1fd7d7286"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.483367 22710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal/instance:
uuid: "3b05d5a6c8a8485d87f873e3999cccd7"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "a6108f5ef19e464c426c1c828b490bc4"
server_key_iv: "22671ff71ca8a1c05d066ec1fd7d7286"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.487321 22710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:39:58.490124 22725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:58.491441 22710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:39:58.491570 22710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal
uuid: "3b05d5a6c8a8485d87f873e3999cccd7"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "a6108f5ef19e464c426c1c828b490bc4"
server_key_iv: "22671ff71ca8a1c05d066ec1fd7d7286"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.491705 22710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:58.501497 22710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:58.504675 22710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:58.504871 22710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:58.512017 22710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40657
I20260927 12:39:58.512310 22777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40657 every 8 connection(s)
I20260927 12:39:58.513141 22710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:39:58.516219 22778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:58.518545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22710
I20260927 12:39:58.518659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-1/wal/instance
I20260927 12:39:58.518920 392 external_mini_cluster.cc:1428] Setting key 8c3aa574dbb46c66684636a8a16321ee
I20260927 12:39:58.520402 22778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:58.534940 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.522839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.0.98.62:43903)
0927 12:39:58.523020 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.523025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.524020 (+ 995us) server_negotiation.cc:407] Connection header received
0927 12:39:58.525068 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.525072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.525148 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.525255 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.526433 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.526959 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.528167 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.528454 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.528547 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.531167 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.531172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.531175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.531209 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.532746 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.533295 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.533299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.533302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.533381 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.533712 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.533715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.533717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.533929 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.534019 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.534610 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.534767 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:39:58.535810 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.523295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43903 (local address 127.0.0.1:41438)
0927 12:39:58.523842 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.523898 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.524842 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.525411 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.525417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.525758 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.526267 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.526283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.527089 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.527092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.528026 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.528037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.528610 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.528614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.528803 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.529373 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.529390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.530967 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.532879 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.532886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.532906 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.533198 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.533464 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.533468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.533470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.533621 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.534036 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.534043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.534354 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.534538 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.534904 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":143,"threads_started":1}
I20260927 12:39:58.537360 22778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:39:58.538311 22703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } attempt: 3
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:39:58.548002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39815
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43903,127.0.98.61:40657,127.0.98.60:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:39:58.548779 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.537907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33384)
0927 12:39:58.538053 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.538065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.538165 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.539161 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.539164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.539350 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.539439 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.539445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.540603 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.540606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.541509 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.541518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.542078 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.542081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.542168 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.542748 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.542760 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.544393 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.547105 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.547111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.547356 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.547627 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.547631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.547719 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.548271 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.548274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.548356 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.548451 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.548600 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 12:39:58.548779 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.538700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33390)
0927 12:39:58.538891 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.538905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.538980 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.539301 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.539305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.539449 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.539559 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.539566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.540686 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.540689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.541224 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.541233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.542067 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.542071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.542179 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.542706 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.542719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.544585 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.547197 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.547203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.547450 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.547721 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.547727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.547829 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.548283 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.548286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.548389 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.548469 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.548602 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 12:39:58.549530 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.538774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.98.61:40657)
0927 12:39:58.539043 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.539049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.539059 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:39:58.539086 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.539088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.539128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.539264 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.539727 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.540562 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.541351 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.541905 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.541998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.544805 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.544809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.544812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.544847 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.547037 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547555 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.547558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.547560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.547635 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547933 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.547936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.547938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.548166 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.549028 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.549167 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.549342 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":214,"spinlock_wait_cycles":6656,"thread_start_us":85,"threads_started":1}
I20260927 12:39:58.549551 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.538020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.98.61:40657)
0927 12:39:58.538366 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.538377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.538396 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:39:58.538441 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.538448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.538665 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.539172 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.539547 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.540475 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.541643 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.541901 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.542082 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.544611 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.544629 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.544653 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.544683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.546959 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547460 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.547464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.547465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.547540 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.547795 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.547797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.547798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.548161 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.549078 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.549151 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.549413 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":226,"spinlock_wait_cycles":3712,"thread_start_us":133,"threads_started":1}
I20260927 12:39:58.551702 22703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:39:58.551836 22778 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:39:58.552551 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.552247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60924)
0927 12:39:58.552417 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.552491 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 12:39:58.552546 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.552248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60920)
0927 12:39:58.552399 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.552465 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 12:39:58.552762 22727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
I20260927 12:39:58.553515 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.553203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60940)
0927 12:39:58.553397 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.553448 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 12:39:58.553581 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.553377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60954)
0927 12:39:58.553478 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.553524 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 12:39:58.553653 22703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
W20260927 12:39:58.553781 22778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
I20260927 12:39:58.584805 22703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } attempt: 1
I20260927 12:39:58.585777 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.585431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60960)
0927 12:39:58.585606 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.585681 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 12:39:58.586757 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.586407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60974)
0927 12:39:58.586591 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.586662 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:39:58.587038 22703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
I20260927 12:39:58.603127 22778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } attempt: 1
I20260927 12:39:58.604171 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.603793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60982)
0927 12:39:58.603983 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.604051 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 12:39:58.604874 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.604640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60988)
0927 12:39:58.604761 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.604809 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:39:58.605027 22778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
I20260927 12:39:58.662250 22703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } attempt: 2
I20260927 12:39:58.663368 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.663025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:60996)
0927 12:39:58.663198 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.663271 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 12:39:58.664263 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.663982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:32776)
0927 12:39:58.664127 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.664183 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:39:58.664422 22703 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
I20260927 12:39:58.673182 22778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } attempt: 2
W20260927 12:39:58.673645 22785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260927 12:39:58.674031 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.673682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:32784)
0927 12:39:58.673866 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.673939 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 12:39:58.674052 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:39:58.674089 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:39:58.674150 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:39:58.674729 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.674512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:32800)
0927 12:39:58.674609 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.674658 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 12:39:58.674885 22778 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39815: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39815: connect: Connection refused (error 111)
W20260927 12:39:58.678800 22785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:39:58.678869 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:39:58.678887 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:39:58.678903 22785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:39:58.678916 22785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:39:58.684567 22785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43903,127.0.98.61:40657,127.0.98.60:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:39:58.685649 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:39:58.686630 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:39:58.695022 22794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:58.695022 22792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:39:58.695044 22791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:39:58.695218 22785 server_base.cc:1034] running on GCE node
I20260927 12:39:58.695858 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:39:58.697073 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:39:58.698246 22785 hybrid_clock.cc:648] HybridClock initialized: now 1790512798698228 us; error 33 us; skew 500 ppm
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:39:58.701143 22785 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:39:58.702023 22785 webserver.cc:460] Webserver started at http://127.0.98.60:37839/ using document root <none> and password file <none>
I20260927 12:39:58.702543 22785 fs_manager.cc:359] Metadata directory not provided
I20260927 12:39:58.702598 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:39:58.702771 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:39:58.704916 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data/instance:
uuid: "9fbb43a9d64a4297bbd1b75df98a298e"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "618c35384b3e84e49e72c69161dfeba7"
server_key_iv: "04a09d08edf9263f679f39425ffb7ff1"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.705497 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal/instance:
uuid: "9fbb43a9d64a4297bbd1b75df98a298e"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "618c35384b3e84e49e72c69161dfeba7"
server_key_iv: "04a09d08edf9263f679f39425ffb7ff1"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.709841 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:39:58.712508 22801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:39:58.713896 22785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:39:58.714042 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal
uuid: "9fbb43a9d64a4297bbd1b75df98a298e"
format_stamp: "Formatted at 2026-09-27 12:39:58 on dist-test-slave-tzgx"
server_key: "618c35384b3e84e49e72c69161dfeba7"
server_key_iv: "04a09d08edf9263f679f39425ffb7ff1"
server_key_version: "encryptionkey@0"
I20260927 12:39:58.714176 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:39:58.729646 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:39:58.733361 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:39:58.733561 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:39:58.741406 22785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39815
I20260927 12:39:58.741684 22853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39815 every 8 connection(s)
I20260927 12:39:58.742554 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:39:58.746380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 22785
I20260927 12:39:58.746464 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:39:58.746510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512512765972-392-0/minicluster-data/master-2/wal/instance
I20260927 12:39:58.746824 392 external_mini_cluster.cc:1428] Setting key 4ba61f126114aeceb458ecbb4bf5c18d
I20260927 12:39:58.752734 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:39:58.763592 22703 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } attempt: 3
I20260927 12:39:58.764896 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.748366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.0.98.60:39815)
0927 12:39:58.749762 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.749790 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.749936 (+ 146us) server_negotiation.cc:407] Connection header received
0927 12:39:58.750881 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.750909 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.751330 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.751835 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.752737 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.753896 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.754882 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.755872 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.756007 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.759032 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.759039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.759052 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.759083 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.761318 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.761777 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.761782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.761788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.761922 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.762386 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.762390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.762392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.762912 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.763187 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.763484 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.763797 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":165,"threads_started":1}
I20260927 12:39:58.765889 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.754562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43903 (local address 127.0.0.1:41452)
0927 12:39:58.754983 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.755027 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.755149 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.755425 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.755431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.755817 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.755959 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.755967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.756839 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.756843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.757976 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.758032 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.758827 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.758834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.758969 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.759531 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.759547 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.761379 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.763291 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.763298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.763304 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.763902 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.764261 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.764264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.764266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.764363 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.765235 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.765241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.765377 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.765523 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.765706 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260927 12:39:58.766074 22778 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } attempt: 3
I20260927 12:39:58.766556 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.754639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.0.98.62:43903)
0927 12:39:58.754807 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.754813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.755106 (+ 293us) server_negotiation.cc:407] Connection header received
0927 12:39:58.755245 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.755249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.755292 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.755360 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.756139 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.756701 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.758254 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.758617 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.758724 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.761552 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.761556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.761558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.761598 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.763175 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.764020 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.764024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.764026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.764147 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.764457 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.764460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.764462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.764687 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.764785 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.765719 (+ 934us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.765849 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:39:58.767803 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:39:58.777681 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.764045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:32808)
0927 12:39:58.764274 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.764288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.764363 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.764811 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.764814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.764991 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.765098 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.765105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.766465 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.766468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.767224 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.767236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.768181 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.768187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.768500 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.769314 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.769334 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.771791 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.774909 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.774913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.774916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.775556 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.776028 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.776031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.776033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.776167 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.776938 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.776941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.777270 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.777353 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.777498 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 12:39:58.777748 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.766820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.0.98.60:39815)
0927 12:39:58.767362 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.767366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.767592 (+ 226us) server_negotiation.cc:407] Connection header received
0927 12:39:58.767630 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.767632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.767677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.768299 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.768390 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.769104 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.770463 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.770829 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.771345 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.774270 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.774274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.774276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.774304 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.775665 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.776289 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.776292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.776294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.776416 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.776721 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.776724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.776726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.776913 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.777412 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.777495 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.777612 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"spinlock_wait_cycles":1792}
I20260927 12:39:58.778236 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.764115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.98.60:39815)
0927 12:39:58.764586 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.764595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.764607 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:39:58.764637 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.764640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.764681 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.764771 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.765206 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.765897 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.767466 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.767778 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.772180 (+ 4402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.772332 (+ 152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.772335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.772338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.772360 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.774246 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.775720 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.775724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.775725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.775899 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.776289 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.776292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.776294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.776814 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.777941 (+ 1127us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.778013 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.778101 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"spinlock_wait_cycles":1920,"thread_start_us":97,"threads_started":1}
I20260927 12:39:58.778743 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.766608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:32816)
0927 12:39:58.767417 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.767432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.767516 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.767804 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.767807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.768003 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.768155 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.768164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.769541 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.769545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.770307 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.770320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.770963 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.770966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.771055 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.771828 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.771843 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.774112 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.775894 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.775897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.775899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.776186 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.776521 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.776525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.776526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.776614 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.777006 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.777010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.777114 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.777199 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.778598 (+ 1399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721}
I20260927 12:39:58.779074 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.767782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.0.98.62:43903)
0927 12:39:58.767890 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.767894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.768194 (+ 300us) server_negotiation.cc:407] Connection header received
0927 12:39:58.768235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.768237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.768282 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.768343 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.769261 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.770008 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.771082 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.771383 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.771477 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.774363 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.774367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.774369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.774400 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.776429 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.776943 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.776947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.776949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.777034 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.777357 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.777360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.777362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.777597 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.778376 (+ 779us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.778702 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.778825 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:39:58.781090 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.769324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.98.61:40657)
0927 12:39:58.769453 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.769457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.769470 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:39:58.769687 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.769692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.769754 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.769810 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.771198 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.772181 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.773418 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.773781 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.773880 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.776229 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.776233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.776236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.776265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.777877 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.779276 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.779280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.779282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.779508 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.779837 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.779841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.779843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.780077 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.780178 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.780799 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.780940 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:39:58.781531 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.768921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40657 (local address 127.0.0.1:33398)
0927 12:39:58.769405 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.769422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.769538 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.770044 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.770048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.770245 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.771040 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.771052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.772319 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.772322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.773170 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.773183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.774068 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.774072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.774171 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.774559 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.774574 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.776037 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.778910 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.778913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.778915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.779148 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.779628 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.779631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.779633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.779728 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.780250 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.780253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.780578 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.780721 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.780909 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365}
I20260927 12:39:58.782449 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b: Bootstrap starting.
I20260927 12:39:58.782470 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:39:58.785053 22778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: Bootstrap starting.
I20260927 12:39:58.785645 22703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:58.786602 22703 log.cc:824] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:39:58.787983 22778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:58.789072 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b: No bootstrap required, opened a new log
I20260927 12:39:58.790011 22778 log.cc:824] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:39:58 dist-test-slave-tzgx krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:39:58.793366 22778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: No bootstrap required, opened a new log
I20260927 12:39:58.794438 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.780302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.98.61:40657)
0927 12:39:58.780641 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.780645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.780659 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:39:58.780695 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.780697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.780741 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.780805 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.782609 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.783354 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.784457 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.784800 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.784875 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.789860 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.789864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.789868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.789901 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.791776 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.792394 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.792398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.792400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.792489 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.792812 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.792816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.792818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.793071 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.793273 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.794102 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.794237 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 12:39:58.794272 22703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.794627 22703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a74f2975ca348c591a953dc6f5f9e4b, State: Initialized, Role: FOLLOWER
I20260927 12:39:58.795176 22703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.796365 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.782953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39815 (local address 127.0.0.1:32832)
0927 12:39:58.783284 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 12:39:58.783298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:39:58.783377 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:39:58.784501 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:39:58.784504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:39:58.784680 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:39:58.784760 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.784767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.785764 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.785767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.786506 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:39:58.786516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.787145 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.787148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:39:58.787219 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.791212 (+ 3993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:39:58.791231 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:39:58.792816 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:39:58.794713 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.794716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.794718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.794951 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.795227 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:39:58.795230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:39:58.795232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:39:58.795299 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:39:58.795805 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:39:58.795808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:39:58.795922 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 12:39:58.796024 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 12:39:58.796195 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260927 12:39:58.796900 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.783130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.0.98.60:39815)
0927 12:39:58.784239 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.784243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.784256 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:39:58.784289 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.784293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.784337 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.784406 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.784881 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.785624 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.786625 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.786940 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.787034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.792961 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.792964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.792967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.792991 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.794604 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.795054 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.795057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.795059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.795143 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.795443 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.795446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.795448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.795659 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.795742 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.796630 (+ 888us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.796749 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1010}
I20260927 12:39:58.797122 22703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:58.799474 22778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.799788 22778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b05d5a6c8a8485d87f873e3999cccd7, State: Initialized, Role: FOLLOWER
I20260927 12:39:58.800442 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: Bootstrap starting.
I20260927 12:39:58.801119 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.800171 22778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.801380 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.802014 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.802225 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.802700 22778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:39:58.802964 22873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:58.803030 22873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:58.803275 22854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: Neither blocks nor log segments found. Creating new log.
I20260927 12:39:58.804226 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:39:58.805020 22854 log.cc:824] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:39:58.806100 22677 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a74f2975ca348c591a953dc6f5f9e4b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41464
I20260927 12:39:58.806375 22677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 12:39:58.806501 22677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:58.807916 22677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.807960 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: No bootstrap required, opened a new log
I20260927 12:39:58.808880 22677 leader_election.cc:288] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [CANDIDATE]: Term 1 election: Requested vote from peers 3b05d5a6c8a8485d87f873e3999cccd7 (127.0.98.61:40657), 9fbb43a9d64a4297bbd1b75df98a298e (127.0.98.60:39815)
I20260927 12:39:58.811682 22752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3b05d5a6c8a8485d87f873e3999cccd7"
I20260927 12:39:58.811975 22752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:39:58.812340 22854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.812602 22854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb43a9d64a4297bbd1b75df98a298e, State: Initialized, Role: FOLLOWER
I20260927 12:39:58.812963 22854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.813405 22752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a74f2975ca348c591a953dc6f5f9e4b in term 1.
I20260927 12:39:58.813912 22653 leader_election.cc:302] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a74f2975ca348c591a953dc6f5f9e4b, 3b05d5a6c8a8485d87f873e3999cccd7; no voters:
I20260927 12:39:58.814381 22862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:39:58.814600 22862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [term 1 LEADER]: Becoming Leader. State: Replica: 1a74f2975ca348c591a953dc6f5f9e4b, State: Running, Role: LEADER
I20260927 12:39:58.815001 22862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } }
I20260927 12:39:58.815403 22854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:39:58.817052 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a74f2975ca348c591a953dc6f5f9e4b. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.817183 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:58.818203 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9fbb43a9d64a4297bbd1b75df98a298e"
I20260927 12:39:58.819249 22828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:39:58.819289 22886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:58.819365 22886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:58.820710 22828 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a74f2975ca348c591a953dc6f5f9e4b in term 1.
I20260927 12:39:58.823841 22904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:39:58.825039 22904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:39:58.824913 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.826068 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.827682 22635 master_runner.cc:263] No leader master found from master 1a74f2975ca348c591a953dc6f5f9e4b
I20260927 12:39:58.827757 22635 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 12:39:58.829290 22910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:39:58.829384 22910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:39:58.834976 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.811201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.0.98.61:40657)
0927 12:39:58.813721 (+ 2520us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.813725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.820154 (+ 6429us) server_negotiation.cc:407] Connection header received
0927 12:39:58.820377 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.820380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.820427 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.820477 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.821563 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.822437 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.823564 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.823858 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.823933 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.831122 (+ 7189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.831127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.831130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.831161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.832970 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.833498 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.833502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.833504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.833593 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.833947 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.833951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.833953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.834179 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.834297 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.834653 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.834796 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2436}
I20260927 12:39:58.838872 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.814491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.98.62:43903)
0927 12:39:58.814608 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.814612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.826348 (+ 11736us) server_negotiation.cc:407] Connection header received
0927 12:39:58.826400 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.826403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.826451 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.826517 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.830733 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.831606 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.832933 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.833282 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.833359 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.835074 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.835078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.835081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.835109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.836781 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.837272 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.837276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.837278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.837369 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.837658 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.837662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.837664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.837882 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.837996 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.838577 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.838700 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:39:58.842332 22828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [term 1 FOLLOWER]: Refusing update from remote peer 1a74f2975ca348c591a953dc6f5f9e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:58.843075 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:58.843966 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:39:58.815077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.0.98.60:39815)
0927 12:39:58.815191 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.815195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.835399 (+ 20204us) server_negotiation.cc:407] Connection header received
0927 12:39:58.835581 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.835584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.835639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.835688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.836506 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.837096 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.838128 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:39:58.838444 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:39:58.838521 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:39:58.840392 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:39:58.840397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:39:58.840399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:39:58.840427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:39:58.842028 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.842549 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.842553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.842555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.842641 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:39:58.842959 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:39:58.842963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:39:58.842965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:39:58.843195 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:39:58.843330 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:39:58.843673 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 12:39:58.843808 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:39:58.847882 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a74f2975ca348c591a953dc6f5f9e4b. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.848012 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.849212 22752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [term 1 FOLLOWER]: Refusing update from remote peer 1a74f2975ca348c591a953dc6f5f9e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:39:58.850095 22862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:39:58.851042 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.851154 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:58.855346 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a74f2975ca348c591a953dc6f5f9e4b. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.855602 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.855764 22917 mvcc.cc:205] Tried to move back new op lower bound from 7333940424049737728 to 7333940423950229504. Current Snapshot: MvccSnapshot[applied={T|T < 7333940424049737728}]
I20260927 12:39:58.857283 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.857419 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.858371 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.858484 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:39:58.859051 22911 mvcc.cc:205] Tried to move back new op lower bound from 7333940424049737728 to 7333940423950229504. Current Snapshot: MvccSnapshot[applied={T|T < 7333940424049737728}]
I20260927 12:39:58.859441 22913 mvcc.cc:205] Tried to move back new op lower bound from 7333940424049737728 to 7333940423950229504. Current Snapshot: MvccSnapshot[applied={T|T < 7333940424049737728}]
I20260927 12:39:58.863075 22904 catalog_manager.cc:1347] Generated new cluster ID: 80050a4483bd44cc813e989e94567909
I20260927 12:39:58.863159 22904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:39:58.900189 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a74f2975ca348c591a953dc6f5f9e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43903 } } peers { permanent_uuid: "3b05d5a6c8a8485d87f873e3999cccd7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40657 } } peers { permanent_uuid: "9fbb43a9d64a4297bbd1b75df98a298e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39815 } } }
I20260927 12:39:58.900381 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b [sys.catalog]: This master's current role is: LEADER
I20260927 12:39:58.902359 22904 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:39:58.904523 22904 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:39:58.920885 22904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a74f2975ca348c591a953dc6f5f9e4b: Generated new TSK 0
I20260927 12:39:58.922004 22904 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:39:58.960054 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.956826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.98.61:40657)
0927 12:39:58.956957 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.956961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.958948 (+ 1987us) server_negotiation.cc:407] Connection header received
0927 12:39:58.959147 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.959151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.959202 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.959264 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.959912 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33438: BlockingRecv error: recv got EOF from 127.0.0.1:33438 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 12:39:58.961442 22859 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.957642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.98.60:39815)
0927 12:39:58.957756 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.957761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.960567 (+ 2806us) server_negotiation.cc:407] Connection header received
0927 12:39:58.960721 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.960724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.960772 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.960820 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.961321 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32842: BlockingRecv error: recv got EOF from 127.0.0.1:32842 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:39:58.962863 22707 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.958344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.98.62:43903)
0927 12:39:58.958461 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.958465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.961917 (+ 3452us) server_negotiation.cc:407] Connection header received
0927 12:39:58.962162 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.962165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.962214 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.962277 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.962747 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41490: BlockingRecv error: recv got EOF from 127.0.0.1:41490 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 12:39:58.967700 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.965963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.98.61:40657)
0927 12:39:58.966078 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.966083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.966618 (+ 535us) server_negotiation.cc:407] Connection header received
0927 12:39:58.966853 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.966857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.966938 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.966991 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.967580 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33454: BlockingRecv error: recv got EOF from 127.0.0.1:33454 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:39:58.973089 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.971356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.0.98.61:40657)
0927 12:39:58.971476 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.971480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.972059 (+ 579us) server_negotiation.cc:407] Connection header received
0927 12:39:58.972317 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.972320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.972373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.972424 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.972965 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33458: BlockingRecv error: recv got EOF from 127.0.0.1:33458 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:39:58.979470 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.977524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.61:40657)
0927 12:39:58.977650 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.977654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.978248 (+ 594us) server_negotiation.cc:407] Connection header received
0927 12:39:58.978610 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.978615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.978678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.978733 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.979342 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33464: BlockingRecv error: recv got EOF from 127.0.0.1:33464 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 12:39:58.987927 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.986136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.0.98.61:40657)
0927 12:39:58.986289 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.986295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.986926 (+ 631us) server_negotiation.cc:407] Connection header received
0927 12:39:58.987132 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.987136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.987191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.987241 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:58.987810 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33470: BlockingRecv error: recv got EOF from 127.0.0.1:33470 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 12:39:59.000593 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:58.998572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.98.61:40657)
0927 12:39:58.998702 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:58.998707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:58.999383 (+ 676us) server_negotiation.cc:407] Connection header received
0927 12:39:58.999609 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:58.999613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:58.999692 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:58.999744 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:59.000445 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33474: BlockingRecv error: recv got EOF from 127.0.0.1:33474 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 12:39:59.021541 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:59.019360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.0.98.61:40657)
0927 12:39:59.019495 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:59.019499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:59.020396 (+ 897us) server_negotiation.cc:407] Connection header received
0927 12:39:59.020640 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:59.020644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:59.020706 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:59.020765 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:59.021411 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33480: BlockingRecv error: recv got EOF from 127.0.0.1:33480 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 12:39:59.057972 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:59.056338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.0.98.61:40657)
0927 12:39:59.056487 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:59.056493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:59.056715 (+ 222us) server_negotiation.cc:407] Connection header received
0927 12:39:59.056941 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:59.056948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:59.057046 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:59.057182 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:59.057839 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33494: BlockingRecv error: recv got EOF from 127.0.0.1:33494 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 12:39:59.125411 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:59.124311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.0.98.61:40657)
0927 12:39:59.124464 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:59.124467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:59.124581 (+ 114us) server_negotiation.cc:407] Connection header received
0927 12:39:59.124711 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:59.124714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:59.124763 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:59.124850 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:59.125260 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33500: BlockingRecv error: recv got EOF from 127.0.0.1:33500 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 12:39:59.256760 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:59.255720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.0.98.61:40657)
0927 12:39:59.255872 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:59.255876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:59.255970 (+ 94us) server_negotiation.cc:407] Connection header received
0927 12:39:59.256131 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:59.256134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:59.256180 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:59.256243 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:59.256642 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33504: BlockingRecv error: recv got EOF from 127.0.0.1:33504 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 12:39:59.516121 22783 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:39:59.515012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.0.98.61:40657)
0927 12:39:59.515169 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:39:59.515174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:39:59.515274 (+ 100us) server_negotiation.cc:407] Connection header received
0927 12:39:59.515391 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:39:59.515394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:39:59.515448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:39:59.515529 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:39:59.515993 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33518: BlockingRecv error: recv got EOF from 127.0.0.1:33518 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:39:59.820804 22886 catalog_manager.cc:1259] Loaded cluster ID: 80050a4483bd44cc813e989e94567909
I20260927 12:39:59.820899 22886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: loading cluster ID for follower catalog manager: success
I20260927 12:39:59.822661 22886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: acquiring CA information for follower catalog manager: success
I20260927 12:39:59.823973 22886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b05d5a6c8a8485d87f873e3999cccd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:39:59.830750 22910 catalog_manager.cc:1259] Loaded cluster ID: 80050a4483bd44cc813e989e94567909
I20260927 12:39:59.830852 22910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: loading cluster ID for follower catalog manager: success
I20260927 12:39:59.832688 22910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: acquiring CA information for follower catalog manager: success
I20260927 12:39:59.834019 22910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fbb43a9d64a4297bbd1b75df98a298e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:40:00.034958 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:00.030476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.0.98.61:40657)
0927 12:40:00.030728 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:00.030732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:00.030750 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:40:00.030873 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:00.030876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:00.030998 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:00.031093 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:00.031496 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.032316 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.033283 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.033528 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.033623 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:00.034024 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:00.034516 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:00.034619 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:00.034726 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 12:40:00.034786 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260927 12:40:00.042239 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:00.038300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.0.98.60:39815)
0927 12:40:00.038537 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:00.038541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:00.038558 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:40:00.038758 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:00.038761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:00.038891 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:00.039009 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:00.039272 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.039808 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.040734 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.040977 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.041065 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:00.041364 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:00.041824 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:00.041914 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:00.041995 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 12:40:00.042051 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260927 12:40:00.048972 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:00.045658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.0.98.60:39815)
0927 12:40:00.045766 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:00.045769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:00.045782 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:40:00.045873 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:00.045877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:00.045922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:00.045981 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:00.046288 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.046938 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.047841 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.048059 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.048214 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:00.048486 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:00.048578 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:00.048653 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:00.048750 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 12:40:00.048816 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:40:00.050141 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:00.045324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.98.62:43903)
0927 12:40:00.045661 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:00.045666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:00.046093 (+ 427us) server_negotiation.cc:407] Connection header received
0927 12:40:00.046204 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:00.046207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:00.046381 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:00.046504 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:00.046718 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.047497 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.048507 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.048849 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.048942 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:00.049363 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:00.049660 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:00.049748 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:00.049902 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 12:40:00.049970 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":86,"threads_started":1}
I20260927 12:40:00.050587 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:00.045346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.98.61:40657)
0927 12:40:00.045501 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:00.045506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:00.046018 (+ 512us) server_negotiation.cc:407] Connection header received
0927 12:40:00.046238 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:00.046241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:00.046297 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:00.046373 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:00.046655 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.047493 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.049340 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:00.049636 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:00.049741 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:00.050153 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:00.050234 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:00.050308 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:00.050400 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 12:40:00.050466 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:40:00.057471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22635
I20260927 12:40:00.067657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22710
I20260927 12:40:00.073167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 22785
2026-09-27T12:40:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3070 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:00 dist-test-slave-tzgx krb5kdc[23004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:00 dist-test-slave-tzgx krb5kdc[23004](info): set up 2 sockets
Sep 27 12:40:00 dist-test-slave-tzgx krb5kdc[23004](info): commencing operation
krb5kdc: starting...
W20260927 12:40:01.068475 392 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:01.207765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39827,127.0.98.61:36199,127.0.98.60:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:01.337208 23017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:40:01.337569 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:01.337607 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:01.337674 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:01.341952 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:01.342036 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:01.342063 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:01.342084 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:01.342106 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:01.348913 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39827,127.0.98.61:36199,127.0.98.60:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:01.350086 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:01.351145 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:01.360803 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:01.360946 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:01.360839 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:01.361788 23017 server_base.cc:1034] running on GCE node
I20260927 12:40:01.362605 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:01.363920 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:01.365128 23017 hybrid_clock.cc:648] HybridClock initialized: now 1790512801365098 us; error 54 us; skew 500 ppm
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:01.368350 23017 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:01.369390 23017 webserver.cc:460] Webserver started at http://127.0.98.62:41491/ using document root <none> and password file <none>
I20260927 12:40:01.369987 23017 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:01.370060 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:01.370271 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:01.372556 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data/instance:
uuid: "1ce7a5209516428386c419dff74a19d5"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "88183bae63e20018b05d25a13476431a"
server_key_iv: "643b7d7244cdcd09734706f48ce87f82"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.373188 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ce7a5209516428386c419dff74a19d5"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "88183bae63e20018b05d25a13476431a"
server_key_iv: "643b7d7244cdcd09734706f48ce87f82"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.377722 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:40:01.380959 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:01.382470 23017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:40:01.382618 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal
uuid: "1ce7a5209516428386c419dff74a19d5"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "88183bae63e20018b05d25a13476431a"
server_key_iv: "643b7d7244cdcd09734706f48ce87f82"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.382753 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:01.396752 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:01.401058 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:01.401283 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:01.409927 23017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39827
I20260927 12:40:01.410322 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39827 every 8 connection(s)
I20260927 12:40:01.411003 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:01.414212 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:01.416002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 23017
I20260927 12:40:01.416196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:01.416450 392 external_mini_cluster.cc:1428] Setting key a232118449c82a329a770f8b1e5c6930
I20260927 12:40:01.418466 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:01.437255 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.421078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39827 (local address 127.0.0.1:58818)
0927 12:40:01.421690 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.421730 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.422639 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.423370 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.423376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.423813 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.424111 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.424334 (+ 223us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.425905 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.425908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.426738 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.426750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.427485 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.427489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.427673 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.428581 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.428600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.430439 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.433403 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.433414 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.433428 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.433747 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.434170 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.434174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.434176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.434308 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.434982 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.434989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.435136 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.435316 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.435739 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":158,"thread_start_us":200,"threads_started":1}
I20260927 12:40:01.437256 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.421078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.0.98.62:39827)
0927 12:40:01.421559 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.421579 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.421848 (+ 269us) server_negotiation.cc:407] Connection header received
0927 12:40:01.422967 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.422972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.423182 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.423353 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.424565 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.425765 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.426857 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.427282 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.427448 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.430765 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.430774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.430793 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.430834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.433219 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.433905 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.433911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.433912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.434057 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.434404 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.434408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.434410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.434856 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.435003 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.435392 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.435665 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":322,"thread_start_us":121,"threads_started":1}
I20260927 12:40:01.440716 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:40:01.441615 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.441322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34696)
0927 12:40:01.441458 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.441539 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 12:40:01.441826 23033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
I20260927 12:40:01.442765 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.442532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34706)
0927 12:40:01.442644 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.442692 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:40:01.442903 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36199: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:40:01.448504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36199
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39827,127.0.98.61:36199,127.0.98.60:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:40:01.492172 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } attempt: 1
I20260927 12:40:01.493027 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.492720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34720)
0927 12:40:01.492886 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.492948 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 12:40:01.494115 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.493742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34730)
0927 12:40:01.493922 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.493979 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260927 12:40:01.494304 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36199: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
I20260927 12:40:01.562598 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } attempt: 2
I20260927 12:40:01.563508 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.563209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34740)
0927 12:40:01.563365 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.563428 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 12:40:01.564364 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.564043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34756)
0927 12:40:01.564173 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.564225 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 12:40:01.564527 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36199: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36199: connect: Connection refused (error 111)
W20260927 12:40:01.567432 23092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:40:01.567766 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:01.567790 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:01.567867 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:01.572688 23092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:01.572777 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:01.572796 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:01.572813 23092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:01.572829 23092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:01.579252 23092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39827,127.0.98.61:36199,127.0.98.60:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:01.580785 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:01.582095 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:01.591718 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:01.591801 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:01.591801 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:01.592011 23092 server_base.cc:1034] running on GCE node
I20260927 12:40:01.592736 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:01.594185 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:01.595400 23092 hybrid_clock.cc:648] HybridClock initialized: now 1790512801595380 us; error 39 us; skew 500 ppm
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:01.599596 23092 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:40:01.600689 23092 webserver.cc:460] Webserver started at http://127.0.98.61:37481/ using document root <none> and password file <none>
I20260927 12:40:01.601285 23092 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:01.601358 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:01.601583 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:01.603773 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data/instance:
uuid: "6b5174c44bb745419e3cc2b4e1cb1c41"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "9b657e2d165a71c30b0e37bbc166ec4d"
server_key_iv: "ac9c5ea9b33a3edc06adc0e80494fb35"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.604396 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal/instance:
uuid: "6b5174c44bb745419e3cc2b4e1cb1c41"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "9b657e2d165a71c30b0e37bbc166ec4d"
server_key_iv: "ac9c5ea9b33a3edc06adc0e80494fb35"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.608505 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:40:01.611521 23107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:01.613008 23092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:40:01.613157 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal
uuid: "6b5174c44bb745419e3cc2b4e1cb1c41"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "9b657e2d165a71c30b0e37bbc166ec4d"
server_key_iv: "ac9c5ea9b33a3edc06adc0e80494fb35"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.613289 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:01.628189 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:01.632177 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:01.632398 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:01.639804 23092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36199
I20260927 12:40:01.640076 23159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36199 every 8 connection(s)
I20260927 12:40:01.640887 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:40:01.643986 23160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:01.646101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 23092
I20260927 12:40:01.646229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-1/wal/instance
I20260927 12:40:01.646592 392 external_mini_cluster.cc:1428] Setting key b14f54073c705be921241d91eb4cc667
I20260927 12:40:01.649075 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 12:40:01.655714 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:40:01.667917 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.656353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34764)
0927 12:40:01.656514 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.656528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.656604 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.657552 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.657555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.657786 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.657915 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.657922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.659456 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.659460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.660238 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.660260 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.661024 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.661027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.661126 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.661662 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.661678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.663514 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.666041 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.666044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.666047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.666296 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.666760 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.666763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.666766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.666861 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.667354 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.667357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.667471 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.667581 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.667722 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 12:40:01.668468 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.652074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39827 (local address 127.0.0.1:58828)
0927 12:40:01.652696 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.652761 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.654001 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.654709 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.654719 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.655234 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.656333 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.656361 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.657662 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.657665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.658830 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.658843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.659519 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.659524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.659713 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.660366 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.660388 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.662310 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.664427 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.664439 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.664454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.664962 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.665332 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.665335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.665337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.665550 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.666234 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.666245 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.666797 (+ 552us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.666982 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.667369 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":128,"threads_started":1}
I20260927 12:40:01.668468 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.656583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.0.98.61:36199)
0927 12:40:01.656988 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.657000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.657029 (+ 29us) server_negotiation.cc:407] Connection header received
0927 12:40:01.657079 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.657086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.657341 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.657532 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.658080 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.659314 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.660366 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.660873 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.660989 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.663677 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.663685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.663703 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.663733 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.665903 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.666537 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.666544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.666546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.666644 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.666957 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.666961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.666962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.667237 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.667371 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.667672 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.667831 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":158,"threads_started":1}
I20260927 12:40:01.668514 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.651693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.0.98.62:39827)
0927 12:40:01.651875 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.651881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.652901 (+ 1020us) server_negotiation.cc:407] Connection header received
0927 12:40:01.654309 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.654312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.654386 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.654475 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.656648 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.657506 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.658977 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.659347 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.659438 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.662468 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.662471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.662473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.662496 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.664301 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.665102 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.665107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.665109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.665222 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.665675 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.665679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.665681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.665949 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.666047 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.668259 (+ 2212us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.668373 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:40:01.670157 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:40:01.670920 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:40:01.671594 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.671317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41784)
0927 12:40:01.671453 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.671516 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 12:40:01.672344 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.672011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41800)
0927 12:40:01.672219 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.672279 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 12:40:01.672477 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:40:01.678985 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.670697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34780)
0927 12:40:01.670820 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.670831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.670907 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.671157 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.671159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.671305 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.671402 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.671410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.672240 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.672243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.672922 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.672932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.673552 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.673558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.673707 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.674549 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.674562 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.675861 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.677507 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.677510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.677511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.677709 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.677997 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.678000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.678001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.678090 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.678516 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.678518 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.678604 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.678686 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.678840 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 12:40:01.679102 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.670835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.0.98.61:36199)
0927 12:40:01.670929 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.670933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.670946 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:40:01.670973 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.670976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.671011 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.671066 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.671504 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.672078 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.673044 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.673327 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.673413 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.676025 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.676028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.676030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.676052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.677413 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.677825 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.677828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.677829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.677914 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.678207 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.678211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.678212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.678418 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.678502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.678793 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.678913 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:40:01.679819 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:40:01.680204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
/tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43557
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39827,127.0.98.61:36199,127.0.98.60:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:40:01.680642 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.680289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41814)
0927 12:40:01.680447 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.680567 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 12:40:01.681270 23111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260927 12:40:01.682211 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.681986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41826)
0927 12:40:01.682101 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.682148 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:40:01.682364 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260927 12:40:01.703682 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 1
I20260927 12:40:01.704787 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.704388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41828)
0927 12:40:01.704625 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.704695 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 12:40:01.705659 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.705417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41842)
0927 12:40:01.705545 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.705595 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 12:40:01.705791 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260927 12:40:01.731673 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 1
I20260927 12:40:01.732622 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.732285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41848)
0927 12:40:01.732451 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.732527 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 12:40:01.733510 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.733189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41860)
0927 12:40:01.733299 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.733428 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:40:01.733686 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260927 12:40:01.780982 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 2
I20260927 12:40:01.782186 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.781610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41876)
0927 12:40:01.781977 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.782066 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 12:40:01.783013 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.782749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41886)
0927 12:40:01.782897 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.782950 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 12:40:01.783164 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
I20260927 12:40:01.801872 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 2
I20260927 12:40:01.802873 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.802522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41892)
0927 12:40:01.802690 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.802763 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 12:40:01.803673 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.803454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41902)
0927 12:40:01.803557 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.803608 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:40:01.803862 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43557: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43557: connect: Connection refused (error 111)
W20260927 12:40:01.804519 23166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:40:01.804839 23166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:01.804881 23166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:01.804945 23166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:01.809894 23166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:01.810003 23166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:01.810034 23166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:01.810323 23166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:01.810366 23166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:01.817219 23166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39827,127.0.98.61:36199,127.0.98.60:43557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:01.818996 23166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:01.820228 23166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:01.829413 23173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:01.829490 23175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:01.829411 23172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:01.829465 23166 server_base.cc:1034] running on GCE node
I20260927 12:40:01.830175 23166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:01.831377 23166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:01.832602 23166 hybrid_clock.cc:648] HybridClock initialized: now 1790512801832579 us; error 41 us; skew 500 ppm
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:01.835553 23166 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:40:01.836658 23166 webserver.cc:460] Webserver started at http://127.0.98.60:44001/ using document root <none> and password file <none>
I20260927 12:40:01.837685 23166 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:01.837924 23166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:01.838248 23166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:01.841974 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data/instance:
uuid: "9f589e7530d94fbca755208fd949f3d0"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "d3361070221e2e9e86c2513fa3372b4d"
server_key_iv: "458d1792e345aac0750d192133f0a20a"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.843164 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal/instance:
uuid: "9f589e7530d94fbca755208fd949f3d0"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "d3361070221e2e9e86c2513fa3372b4d"
server_key_iv: "458d1792e345aac0750d192133f0a20a"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.848820 23166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 12:40:01.852013 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:01.854053 23166 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:40:01.854241 23166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal
uuid: "9f589e7530d94fbca755208fd949f3d0"
format_stamp: "Formatted at 2026-09-27 12:40:01 on dist-test-slave-tzgx"
server_key: "d3361070221e2e9e86c2513fa3372b4d"
server_key_iv: "458d1792e345aac0750d192133f0a20a"
server_key_version: "encryptionkey@0"
I20260927 12:40:01.854382 23166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:01.865087 23166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:01.868400 23166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:01.868603 23166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:01.875988 23166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43557
I20260927 12:40:01.876350 23234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43557 every 8 connection(s)
I20260927 12:40:01.877075 23166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:40:01.877806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu as pid 23166
I20260927 12:40:01.877939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRFp9Fb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512512765972-392-0/minicluster-data/master-2/wal/instance
I20260927 12:40:01.878309 392 external_mini_cluster.cc:1428] Setting key f91c3a5a083404b4ace87b15891d0167
I20260927 12:40:01.880589 23235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:01.882326 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 3
I20260927 12:40:01.885128 23235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:40:01.893337 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.882777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41918)
0927 12:40:01.882901 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.882916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.882992 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.883599 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.883601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.883774 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.883876 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.883883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.884758 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.884761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.885520 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.885532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.886114 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.886118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.886213 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.886694 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.886708 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.888449 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.891043 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.891065 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.891069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.891351 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.891752 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.891755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.891757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.891845 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.892733 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.892737 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.892854 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.892973 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.893137 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:01.894845 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.879783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.0.98.60:43557)
0927 12:40:01.880309 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.880329 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.880368 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:40:01.881088 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.881105 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.881460 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.881783 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.882743 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.884290 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.885266 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.885814 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.885963 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.889016 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.889019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.889021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.889044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.890980 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.891459 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.891465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.891472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.891679 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.891968 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.891971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.891972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.892600 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.892775 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.894023 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.894141 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":349,"spinlock_wait_cycles":16640,"thread_start_us":139,"threads_started":1}
I20260927 12:40:01.894872 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.883055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.0.98.60:43557)
0927 12:40:01.883383 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.883389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.883400 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:40:01.883430 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.883432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.883465 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.883550 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.883983 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.884636 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.885661 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.885983 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.886421 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.888616 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.888623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.888635 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.888667 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.890938 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.891459 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.891465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.891472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.891672 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.891944 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.891947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.891948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.892600 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.892731 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.893124 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.893390 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":268,"spinlock_wait_cycles":12032,"thread_start_us":145,"threads_started":1}
I20260927 12:40:01.895035 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } attempt: 3
I20260927 12:40:01.900123 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.887325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.0.98.62:39827)
0927 12:40:01.887448 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.887452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.887925 (+ 473us) server_negotiation.cc:407] Connection header received
0927 12:40:01.888145 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.888148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.888198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.888272 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.889221 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.889958 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.891176 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.891522 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.891608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.894938 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.894942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.894945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.894973 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.897982 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.898591 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.898594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.898596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.898692 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.898964 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.898967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.898969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.899243 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.899346 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.899810 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.899938 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:40:01.900736 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.887316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39827 (local address 127.0.0.1:58840)
0927 12:40:01.887786 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.887842 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.887978 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.888346 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.888353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.888904 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.889079 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.889089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.890081 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.890085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.891082 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.891090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.891628 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.891631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.891771 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.892264 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.892278 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.894682 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.898124 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.898133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.898139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.898481 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.898772 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.898775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.898777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.898873 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.899404 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.899411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.899546 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.900329 (+ 783us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.900559 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260927 12:40:01.901043 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5: Bootstrap starting.
I20260927 12:40:01.903396 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:01.903947 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.895600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.0.98.60:43557)
0927 12:40:01.895749 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.895753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.895765 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:40:01.895848 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.895850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.895894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.896729 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.896829 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.897650 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.898726 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.899049 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.899142 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.901170 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.901174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.901177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.901202 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.902449 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.902835 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.902839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.902841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.902908 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.903161 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.903164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.903166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.903343 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.903426 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.903693 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.903800 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:40:01.903946 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.895484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41926)
0927 12:40:01.895677 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.895692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.895774 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.896027 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.896030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.896362 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.896598 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.896609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.897817 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.897820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.898626 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.898637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.899184 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.899187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.899268 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.899687 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.899701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.901000 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.902558 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.902561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.902564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.902749 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.902987 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.902990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.902992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.903077 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.903431 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.903434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.903535 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.903631 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.903779 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 12:40:01.905153 23085 log.cc:824] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:01.905165 23235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:40:01.907464 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: Bootstrap starting.
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:40:01.910369 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5: No bootstrap required, opened a new log
I20260927 12:40:01.910478 23160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:01.911399 23160 log.cc:824] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:01.914314 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.905909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.0.98.61:36199)
0927 12:40:01.906009 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.906013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.906026 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:40:01.906061 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.906064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.906105 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.906172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.906619 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.907114 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.907902 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.908180 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.908270 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.910262 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.910266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.910268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.910290 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.911856 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.912489 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.912493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.912495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.912581 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.912920 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.912924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.912926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.913195 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.913302 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.913971 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.914094 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:40:01.914191 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: No bootstrap required, opened a new log
I20260927 12:40:01.914333 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.905668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36199 (local address 127.0.0.1:34794)
0927 12:40:01.905798 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.905812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.905882 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.906270 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.906272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.906403 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.906496 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.906501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.907217 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.907220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.907801 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.907810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.908390 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.908393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.908490 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.908933 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.908949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.910133 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.911973 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.911977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.911979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.912366 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.912706 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.912710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.912713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.912812 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.913350 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.913353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.913459 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.913563 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.913716 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 12:40:01.915014 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.915298 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce7a5209516428386c419dff74a19d5, State: Initialized, Role: FOLLOWER
I20260927 12:40:01.915673 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.916375 23235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:40:01.918866 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.904230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.98.62:39827)
0927 12:40:01.904342 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.904347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.904823 (+ 476us) server_negotiation.cc:407] Connection header received
0927 12:40:01.904863 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.904866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.904917 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.904990 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.906203 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.906932 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.907959 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.908293 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.908383 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.914479 (+ 6096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.914483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.914486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.914517 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.916317 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.917271 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.917276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.917278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.917380 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.917667 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.917670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.917672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.917909 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.918109 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.918569 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.918699 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:40:01.919842 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:40:01.926472 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.916867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43557 (local address 127.0.0.1:41932)
0927 12:40:01.917237 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:01.917251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:01.917320 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:01.917745 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:01.917748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:01.917911 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:01.918010 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.918017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.918968 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.918972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.919717 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:01.919727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.920373 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.920376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:01.920503 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.920974 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:01.920989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:01.922536 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:01.924656 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.924659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.924662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.924901 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.925185 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:01.925188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:01.925190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:01.925270 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:01.925705 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:01.925707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:01.925798 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:40:01.925895 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 12:40:01.926153 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260927 12:40:01.927574 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.917098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.0.98.60:43557)
0927 12:40:01.917394 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.917398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.917416 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:40:01.917484 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.917487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.917567 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.917730 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.918114 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.918853 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.919825 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.920269 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.920369 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.922705 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.922709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.922711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.922754 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.924529 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.925003 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.925006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.925008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.925100 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.925353 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.925355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.925357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.925577 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.925671 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.927179 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.927400 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33}
W20260927 12:40:01.930945 23254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:40:01.931039 23254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:40:01.931100 23235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: Bootstrap starting.
I20260927 12:40:01.928232 23017 master_runner.cc:186] Error getting master registration for 127.0.98.61:36199: OK, instance_id { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" instance_seqno: 1790512801637701 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:40:01.931656 23017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:40:01.931886 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:01.931999 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 12:40:01 dist-test-slave-tzgx krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:40:01.934324 23160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.934584 23160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5174c44bb745419e3cc2b4e1cb1c41, State: Initialized, Role: FOLLOWER
I20260927 12:40:01.934834 23235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:01.934944 23160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.935901 23235 log.cc:824] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:01.936534 23160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:01.936961 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.925253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.0.98.61:36199)
0927 12:40:01.925417 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.925422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.925436 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:40:01.925478 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.925481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.925527 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.927071 (+ 1544us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.927232 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.928257 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.929377 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.929729 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.929821 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.933003 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.933007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.933010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.933045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.934651 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.935248 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.935253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.935254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.935342 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.935623 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.935626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.935628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.935871 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.936057 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.936728 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.936846 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:40:01.940743 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 12:40:01.942497 23268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:40:01.942576 23268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:40:01.942329 23058 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1ce7a5209516428386c419dff74a19d5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58850
I20260927 12:40:01.943120 23058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 12:40:01.943255 23058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:01.944645 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:01.944679 23058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.944777 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:40:01.946295 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ce7a5209516428386c419dff74a19d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41"
I20260927 12:40:01.946548 23133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:01.946605 23235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: No bootstrap required, opened a new log
I20260927 12:40:01.948434 23133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ce7a5209516428386c419dff74a19d5 in term 1.
I20260927 12:40:01.949036 23033 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ce7a5209516428386c419dff74a19d5, 6b5174c44bb745419e3cc2b4e1cb1c41; no voters:
I20260927 12:40:01.949386 23243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:01.949673 23243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce7a5209516428386c419dff74a19d5, State: Running, Role: LEADER
I20260927 12:40:01.949728 23058 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [CANDIDATE]: Term 1 election: Requested vote from peers 6b5174c44bb745419e3cc2b4e1cb1c41 (127.0.98.61:36199), 9f589e7530d94fbca755208fd949f3d0 (127.0.98.60:43557)
I20260927 12:40:01.950075 23243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.950414 23235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.950647 23235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f589e7530d94fbca755208fd949f3d0, State: Initialized, Role: FOLLOWER
I20260927 12:40:01.950977 23235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } }
I20260927 12:40:01.952107 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce7a5209516428386c419dff74a19d5. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:01.952244 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:01.952644 23235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:01.953096 23209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ce7a5209516428386c419dff74a19d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f589e7530d94fbca755208fd949f3d0"
I20260927 12:40:01.953353 23209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:01.954802 23209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ce7a5209516428386c419dff74a19d5 in term 1.
I20260927 12:40:01.955061 23092 master_runner.cc:186] Error getting master registration for 127.0.98.60:43557: OK, instance_id { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" instance_seqno: 1790512801873845 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:40:01.958750 23092 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:40:01.960981 23277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:40:01.962358 23277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:01.964864 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:01.965157 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:40:01.970626 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.952161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.0.98.61:36199)
0927 12:40:01.952277 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.952282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.960867 (+ 8585us) server_negotiation.cc:407] Connection header received
0927 12:40:01.961068 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.961072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.961125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.961179 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.961969 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.962693 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.965081 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.965564 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.965630 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.967145 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.967149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.967152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.967178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.968647 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.969106 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.969110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.969112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.969194 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.969484 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.969488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.969490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.969765 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.969907 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.970316 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.970454 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 12:40:01.974586 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.959416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.0.98.62:39827)
0927 12:40:01.959555 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.959561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.964406 (+ 4845us) server_negotiation.cc:407] Connection header received
0927 12:40:01.964516 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.964519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.964573 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.964657 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.967534 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.968310 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.969449 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.969696 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.969789 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.971054 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.971058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.971061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.971088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.972822 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.973203 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.973207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.973209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.973414 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.973742 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.973756 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.973757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.973948 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.974084 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.974298 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.974418 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 12:40:01.979445 23289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 12:40:01.981567 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:01.960215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.98.60:43557)
0927 12:40:01.960340 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:01.960343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:01.971585 (+ 11242us) server_negotiation.cc:407] Connection header received
0927 12:40:01.971766 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:01.971770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:01.971824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:01.971875 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:01.972934 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.973716 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.975230 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:01.975583 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:01.975663 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:01.977427 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:01.977431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:01.977434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:01.977462 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:01.979033 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.979678 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.979686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.979688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.979914 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:01.980530 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:01.980535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:01.980537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:01.980798 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:01.980917 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:01.981273 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 12:40:01.981407 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 12:40:01.983122 23289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:40:01.989547 23133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [term 1 FOLLOWER]: Refusing update from remote peer 1ce7a5209516428386c419dff74a19d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:01.989621 23209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [term 1 FOLLOWER]: Refusing update from remote peer 1ce7a5209516428386c419dff74a19d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:01.990631 23243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:01.991411 23270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 12:40:01.996490 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce7a5209516428386c419dff74a19d5. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:01.996636 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:40:01.998041 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce7a5209516428386c419dff74a19d5. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:01.998205 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:40:02.002830 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:02.003014 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:02.003155 23292 mvcc.cc:205] Tried to move back new op lower bound from 7333940436941602816 to 7333940436791504896. Current Snapshot: MvccSnapshot[applied={T|T < 7333940436941602816}]
I20260927 12:40:02.004160 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:02.004310 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:40:02.006040 23296 mvcc.cc:205] Tried to move back new op lower bound from 7333940436941602816 to 7333940436791504896. Current Snapshot: MvccSnapshot[applied={T|T < 7333940436941602816}]
I20260927 12:40:02.010349 23277 catalog_manager.cc:1347] Generated new cluster ID: 37a82c880b6b4f7da9faa467fda8ac46
I20260927 12:40:02.010447 23277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:02.019660 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:02.019846 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:02.021278 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ce7a5209516428386c419dff74a19d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ce7a5209516428386c419dff74a19d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39827 } } peers { permanent_uuid: "6b5174c44bb745419e3cc2b4e1cb1c41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36199 } } peers { permanent_uuid: "9f589e7530d94fbca755208fd949f3d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43557 } } }
I20260927 12:40:02.021406 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:40:02.045410 23277 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:02.046779 23277 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:02.056244 23277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce7a5209516428386c419dff74a19d5: Generated new TSK 0
I20260927 12:40:02.057296 23277 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:40:02.105127 23088 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.102132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.0.98.62:39827)
0927 12:40:02.102276 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.102280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.103639 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 12:40:02.104136 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.104140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.104200 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.104255 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.104967 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58866: BlockingRecv error: recv got EOF from 127.0.0.1:58866 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 12:40:02.106669 23239 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.102936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.0.98.60:43557)
0927 12:40:02.103058 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.103062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.104026 (+ 964us) server_negotiation.cc:407] Connection header received
0927 12:40:02.105546 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.105550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.105609 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.105660 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.106456 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41954: BlockingRecv error: recv got EOF from 127.0.0.1:41954 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 12:40:02.109912 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.108399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.98.61:36199)
0927 12:40:02.108533 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.108537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.108894 (+ 357us) server_negotiation.cc:407] Connection header received
0927 12:40:02.109073 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.109076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.109127 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.109180 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.109765 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34816: BlockingRecv error: recv got EOF from 127.0.0.1:34816 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 12:40:02.115108 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.113200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.0.98.61:36199)
0927 12:40:02.113315 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.113319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.113838 (+ 519us) server_negotiation.cc:407] Connection header received
0927 12:40:02.114068 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.114072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.114148 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.114208 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.114979 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34822: BlockingRecv error: recv got EOF from 127.0.0.1:34822 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 12:40:02.121451 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.119560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.0.98.61:36199)
0927 12:40:02.119682 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.119687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.120324 (+ 637us) server_negotiation.cc:407] Connection header received
0927 12:40:02.120560 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.120564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.120628 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.120693 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.121323 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34838: BlockingRecv error: recv got EOF from 127.0.0.1:34838 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 12:40:02.128151 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.126292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.98.61:36199)
0927 12:40:02.126457 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.126462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.127053 (+ 591us) server_negotiation.cc:407] Connection header received
0927 12:40:02.127274 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.127278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.127336 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.127391 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.127987 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34842: BlockingRecv error: recv got EOF from 127.0.0.1:34842 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 12:40:02.138268 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.136404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.0.98.61:36199)
0927 12:40:02.136566 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.136571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.137200 (+ 629us) server_negotiation.cc:407] Connection header received
0927 12:40:02.137415 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.137419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.137476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.137530 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.138139 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34854: BlockingRecv error: recv got EOF from 127.0.0.1:34854 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 12:40:02.151418 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.149426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.0.98.61:36199)
0927 12:40:02.149559 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.149564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.150258 (+ 694us) server_negotiation.cc:407] Connection header received
0927 12:40:02.150500 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.150504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.150564 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.150620 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.151283 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34866: BlockingRecv error: recv got EOF from 127.0.0.1:34866 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 12:40:02.172919 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.170674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.0.98.61:36199)
0927 12:40:02.170803 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.170807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.171430 (+ 623us) server_negotiation.cc:407] Connection header received
0927 12:40:02.171660 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.171664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.171722 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.171775 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.172788 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34870: BlockingRecv error: recv got EOF from 127.0.0.1:34870 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 12:40:02.209581 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.208433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.98.61:36199)
0927 12:40:02.208621 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.208674 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.208691 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:40:02.208841 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.208843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.208898 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.208966 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.209459 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34874: BlockingRecv error: recv got EOF from 127.0.0.1:34874 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 12:40:02.276811 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.275673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.98.61:36199)
0927 12:40:02.275826 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.275831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.275924 (+ 93us) server_negotiation.cc:407] Connection header received
0927 12:40:02.276074 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.276078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.276155 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.276229 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.276687 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34878: BlockingRecv error: recv got EOF from 127.0.0.1:34878 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 12:40:02.408264 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.406957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.98.61:36199)
0927 12:40:02.407132 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.407136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.407248 (+ 112us) server_negotiation.cc:407] Connection header received
0927 12:40:02.407351 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.407354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.407406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.407491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.408045 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34892: BlockingRecv error: recv got EOF from 127.0.0.1:34892 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 12:40:02.667680 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:02.666483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.0.98.61:36199)
0927 12:40:02.666636 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:02.666640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:02.666655 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:40:02.666877 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:02.666880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:02.666932 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:02.667009 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:02.667559 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34908: BlockingRecv error: recv got EOF from 127.0.0.1:34908 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:40:02.943879 23268 catalog_manager.cc:1259] Loaded cluster ID: 37a82c880b6b4f7da9faa467fda8ac46
I20260927 12:40:02.943964 23268 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: loading cluster ID for follower catalog manager: success
I20260927 12:40:02.945719 23268 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: acquiring CA information for follower catalog manager: success
I20260927 12:40:02.946954 23268 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b5174c44bb745419e3cc2b4e1cb1c41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:40:02.984549 23289 catalog_manager.cc:1259] Loaded cluster ID: 37a82c880b6b4f7da9faa467fda8ac46
I20260927 12:40:02.984637 23289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: loading cluster ID for follower catalog manager: success
I20260927 12:40:02.986334 23289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: acquiring CA information for follower catalog manager: success
I20260927 12:40:02.987564 23289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f589e7530d94fbca755208fd949f3d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:40:03.185952 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:03.181923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.0.98.61:36199)
0927 12:40:03.182172 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:03.182177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:03.182357 (+ 180us) server_negotiation.cc:407] Connection header received
0927 12:40:03.182514 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:03.182516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:03.182668 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:03.182772 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:03.183061 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.183584 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.184480 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.184703 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.184792 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:03.185101 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:03.185547 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:03.185654 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:03.185732 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 12:40:03.185790 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260927 12:40:03.194257 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:03.189594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.98.60:43557)
0927 12:40:03.189916 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:03.189921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:03.189941 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:40:03.190084 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:03.190087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:03.190286 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:03.190416 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:03.190791 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.191382 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.192405 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.192679 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.192775 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:03.193135 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:03.193711 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:03.193917 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:03.193983 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 12:40:03.194053 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260927 12:40:03.202795 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:03.197781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.0.98.61:36199)
0927 12:40:03.197947 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:03.197952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:03.198343 (+ 391us) server_negotiation.cc:407] Connection header received
0927 12:40:03.198695 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:03.198698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:03.198762 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:03.198820 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:03.199278 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.199978 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.201381 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.201665 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.201768 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:03.202149 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:03.202281 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:03.202340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:03.202526 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 12:40:03.202606 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:40:03.203414 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:03.198467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.98.60:43557)
0927 12:40:03.198601 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:03.198606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:03.198684 (+ 78us) server_negotiation.cc:407] Connection header received
0927 12:40:03.198780 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:03.198783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:03.198840 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:03.198924 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:03.200184 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.200965 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.202205 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.202463 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.202560 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:03.202914 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:03.203015 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:03.203059 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:03.203213 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 12:40:03.203280 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:40:03.203579 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:03.197729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.0.98.62:39827)
0927 12:40:03.197957 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:03.197963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:03.198242 (+ 279us) server_negotiation.cc:407] Connection header received
0927 12:40:03.198527 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:03.198531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:03.198700 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:03.198802 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:03.199324 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.199931 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.201580 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:03.201930 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:03.202025 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:03.202440 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:03.202872 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:03.202974 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:03.203320 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 12:40:03.203410 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260927 12:40:03.207806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 23017
I20260927 12:40:03.214303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 23092
I20260927 12:40:03.221388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRFp9Fb/build/debug/bin/kudu with pid 23166
2026-09-27T12:40:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3150 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6220 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (290449 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST